lists.starlingx.io
Sign In Sign Up
Manage this list Sign In Sign Up

Keyboard Shortcuts

Thread View

  • j: Next unread message
  • k: Previous unread message
  • j a: Jump to all threads
  • j l: Jump to MailingList overview

Starlingx-discuss

Thread Start a new thread
Download
Threads by month
  • ----- 2025 -----
  • May
  • April
  • March
  • February
  • January
  • ----- 2024 -----
  • December
  • November
  • October
  • September
  • August
  • July
  • June
  • May
  • April
  • March
  • February
  • January
  • ----- 2023 -----
  • December
  • November
  • October
  • September
  • August
  • July
  • June
  • May
  • April
  • March
  • February
  • January
  • ----- 2022 -----
  • December
  • November
  • October
  • September
  • August
  • July
  • June
  • May
  • April
  • March
  • February
  • January
  • ----- 2021 -----
  • December
  • November
  • October
  • September
  • August
  • July
  • June
  • May
  • April
  • March
  • February
  • January
  • ----- 2020 -----
  • December
  • November
  • October
  • September
  • August
  • July
  • June
  • May
  • April
  • March
  • February
  • January
  • ----- 2019 -----
  • December
  • November
  • October
  • September
  • August
  • July
  • June
  • May
  • April
  • March
  • February
  • January
  • ----- 2018 -----
  • December
  • November
  • October
  • September
  • August
  • July
  • June
starlingx-discuss@lists.starlingx.io

February 2019

  • 88 participants
  • 217 discussions
[Starlingx-discuss] [Containers] Minimum memory requirements for containerized environment.
by Cordoba Malibran, Erich 01 Feb '19

01 Feb '19
Hi, After two days of having a containerized system up (without running any edge application, just the steps in the wiki) I started to see some Out of Memory errors like this: Out of memory: Kill process 17906 (kube-apiserver) score 1017 or sacrifice child Killed process 17906 (kube-apiserver) total-vm:460684kB, anon-rss:335088kB, file-rss:0kB, shmem-rss:0kB I have in my 16 GB in my virtual environment as the wiki says, but I'm wondering if that should be enough and this behavior could be some kind of memory leak or the minimum requirement would be a higher memory. Thanks! -Erich Here is some additional information: controller-0:~$ uptime 16:01:40 up 2 days, 22:30, 2 users, load average: 93.28, 45.36, 36.84 controller-0:~$ free -h total used free shared buff/cache available Mem: 17G 12G 206M 62M 4.8G 1.8G Swap: 0B 0B 0B controller-0:~$ top top - 16:00:10 up 2 days, 22:28, 2 users, load average: 12.57, 22.80, 29.58 Tasks: 705 total, 2 running, 703 sleeping, 0 stopped, 0 zombie %Cpu(s): 30.3 us, 18.3 sy, 0.1 ni, 27.6 id, 16.5 wa, 0.0 hi, 4.8 si, 2.4 st KiB Mem : 18330568 total, 247252 free, 13073324 used, 5009992 buff/cache KiB Swap: 0 total, 0 free, 0 used. 1925172 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18306 root 10 -10 2333224 185064 13216 S 100.3 1.0 4222:53 ovs-vswitchd 71 root 20 0 0 0 0 D 26.8 0.0 18:57.26 kswapd0 7202 root 20 0 1657284 311532 3400 S 19.9 1.7 335:52.58 kubelet 16311 root 20 0 460276 330576 2884 S 12.6 1.8 0:40.81 kube-apiserver 4291 root 20 0 66832 12224 2024 R 12.3 0.1 0:00.37 heat-manage 1116 root 20 0 1362488 151664 1728 S 11.6 0.8 265:32.20 dockerd 3 root 20 0 0 0 0 S 9.9 0.0 12:20.95 ksoftirqd/0 1 root 20 0 542540 354140 1436 S 7.9 1.9 12:23.91 systemd 21081 42424 20 0 626124 122172 1476 S 7.3 0.7 2:28.49 apache2 3795 999 20 0 5168964 258240 0 S 5.3 1.4 17:06.77 mysqld 26085 999 20 0 4965476 153132 0 S 4.6 0.8 54:16.75 beam.smp 16717 root 20 0 47140 13592 0 S 4.0 0.1 0:16.91 calico-felix 673 root 20 0 50868 12528 512 S 3.6 0.1 0:01.02 kube-scheduler 20893 root 20 0 1049800 201616 0 S 3.3 1.1 40:30.02 ceph-osd 10666 root 20 0 10.7g 112996 868 S 3.0 0.6 46:28.91 etcd 14146 root 20 0 45972 11968 0 S 3.0 0.1 10:09.31 kube-proxy 3435 root 20 0 142216 9708 0 S 2.3 0.1 8:28.87 coredns 19481 root 20 0 324232 112556 1404 S 2.3 0.6 13:20.23 nova-conductor 1162 influxdb 20 0 674068 9604 1020 S 2.0 0.1 8:44.77 influxd 1358 root 20 0 3469568 25344 0 S 2.0 0.1 13:21.31 docker-containe 13365 42424 20 0 2642820 104684 0 S 2.0 0.6 91:03.49 cinder-volume 16108 root 20 0 41044 12812 0 S 2.0 0.1 6:22.57 confd 18862 root 20 0 321820 109656 832 S 2.0 0.6 4:51.59 nova-scheduler 22478 root 20 0 248220 84272 608 S 2.0 0.5 82:31.64 neutron-dhcp-ag 402 root 20 0 60372 29536 32 S 1.7 0.2 0:45.64 nginx-ingress-c 12942 42424 20 0 892092 94848 572 S 1.7 0.5 86:12.08 cinder-backup 7367 root 18 -2 480460 9792 2436 S 1.3 0.1 18:51.83 sm 10762 root 20 0 2297792 116944 0 S 1.3 0.6 11:23.95 nova-compute 12122 root 20 0 142216 10148 296 S 1.3 0.1 8:23.28 coredns 15942 root 20 0 388912 61312 2716 S 1.3 0.3 28:51.34 fm-api 19825 root 19 -1 3313564 52676 1432 S 1.3 0.3 0:05.07 beam.smp 20357 sysinv 19 -1 382232 79644 2948 S 1.3 0.4 0:08.44 sysinv-api 313 root 0 -20 0 0 0 S 1.0 0.0 1:03.17 kworker/0:1H 720 dbus 20 0 60452 1696 896 S 1.0 0.0 3:50.63 dbus-daemon 4073 root 20 0 113788 1684 1092 S 1.0 0.0 0:00.03 ceph 4245 root 20 0 11828 776 440 S 1.0 0.0 0:00.03 heat-engine-cle 4991 root 20 0 408012 39660 304 S 1.0 0.2 5:57.15 ceph-mon 9171 root 20 0 0 0 0 S 1.0 0.0 0:01.48 kworker/0:47 12036 42424 20 0 666056 97616 448 S 1.0 0.5 26:53.04 cinder-volume 14554 root 20 0 59324 31316 0 S 1.0 0.2 7:19.50 nginx-ingress-c 18727 wrsroot 20 0 160588 2428 1100 S 1.0 0.0 0:02.46 top 20515 42424 20 0 678900 110492 428 S 1.0 0.6 26:26.14 cinder-api 20571 42424 20 0 627436 94572 312 S 1.0 0.5 26:41.77 glance-api 8 root 20 0 0 0 0 S 0.7 0.0 9:28.58 rcu_preempt 4429 snmpd 19 -1 245388 4620 480 S 0.7 0.0 0:08.51 snmpd
1 0
0 0
Re: [Starlingx-discuss] Installation problem: Configuration failed: Failed to update hiera configuration
by Marcel Schaible 01 Feb '19

01 Feb '19
Additional information: We are using the following iso: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190109T16… Marcel > ------------------------------ > > Message: 2 > Date: Fri, 1 Feb 2019 16:31:35 +0100 (CET) > From: Marcel Schaible <marcel(a)schaible-consulting.de> > To: starlingx-discuss(a)lists.starlingx.io > Subject: [Starlingx-discuss] Installation problem: Configuration > failed: Failed to update hiera configuration > Message-ID: <1484346064.187533.1549035095571(a)communicator.strato.com> > Content-Type: text/plain; charset=UTF-8 > > Hi, > > during installation when appling the configuration I'll get an error message from hiera: > > Configuration failed: Failed to update hiera configuration > > Configuration log and apply_manifet.log is attached below. > > Any idea or hint? > > Thanks > > Marcel > > > > localhost:~# config_controller > System Configuration > ==================== > Enter Q at any prompt to abort... > > System date and time: > --------------------- > > The system date and time must be set now. Note that UTC time must be used and > that the date and time must be set as accurately as possible, even if NTP/PTP is > to be configured later. > > Current system date and time (UTC): 2019-02-01 15:15:25 > > Is the current date and time correct? [y/n]: y > Current system date and time will be used. > > System timezone: > ---------------- > > The system timezone must be set now. The timezone must be a valid timezone from > /usr/share/zoneinfo (e.g. UTC, Asia/Hong_Kong, etc...) > > Please input the timezone[UTC]:Europe/Berlin > > System Configuration: > --------------------- > > System mode. Available options are: > > 1) duplex-direct - two node redundant configuration. Management and > infrastructure networks are directly connected to peer ports > 2) duplex - two node redundant configuration. > 3) simplex - single node non-redundant configuration. > System mode [duplex-direct]: > > PXEBoot Network: > ---------------- > > The PXEBoot network is used for initial booting and installation of each node. > IP addresses on this network are reachable only within the data center. > > The default configuration combines the PXEBoot network and the management > network. If a separate PXEBoot network is used, it will share the management > interface, which requires the management network to be placed on a VLAN. > > Configure a separate PXEBoot network [y/N]: > Aborting configuration > localhost:~# config_controller > System Configuration > ==================== > Enter Q at any prompt to abort... > > System date and time: > --------------------- > > The system date and time must be set now. Note that UTC time must be used and > that the date and time must be set as accurately as possible, even if NTP/PTP is > to be configured later. > > Current system date and time (UTC): 2019-02-01 15:15:40 > > Is the current date and time correct? [y/n]: y > Current system date and time will be used. > > System timezone: > ---------------- > > The system timezone must be set now. The timezone must be a valid timezone from > /usr/share/zoneinfo (e.g. UTC, Asia/Hong_Kong, etc...) > > Please input the timezone[UTC]:Europe/Berlin > > System Configuration: > --------------------- > > System mode. Available options are: > > 1) duplex-direct - two node redundant configuration. Management and > infrastructure networks are directly connected to peer ports > 2) duplex - two node redundant configuration. > 3) simplex - single node non-redundant configuration. > System mode [duplex-direct]: 1 > > PXEBoot Network: > ---------------- > > The PXEBoot network is used for initial booting and installation of each node. > IP addresses on this network are reachable only within the data center. > > The default configuration combines the PXEBoot network and the management > network. If a separate PXEBoot network is used, it will share the management > interface, which requires the management network to be placed on a VLAN. > > Configure a separate PXEBoot network [y/N]: N > > Management Network: > ------------------- > > The management network is used for internal communication between platform > components. IP addresses on this network are reachable only within the data > center. > > A management bond interface provides redundant connections for the management > network. > It is strongly recommended to configure Management interface link > aggregation, for All-in-one duplex-direct. > > Management interface link aggregation [y/N]: y > Management interface [bond0]: > Management interface MTU [1500]: > > Specify one of the bonding policies. Possible values are: > 1) 802.3ad (LACP) policy > 2) Active-backup policy > > Management interface bonding policy [802.3ad]: > A maximum of 2 physical interfaces can be attached to the management interface. > > First management interface member [enp3s0f1]: enp8s20f4 > Second management interface member []: enp8s20f5 > Management subnet [192.168.204.0/24]: 172.27.1.0/24 > Use entire management subnet [Y/n]: Y > > IP addresses can be assigned to hosts dynamically or a static IP address can be > specified for each host. This choice applies to both the management network and > infrastructure network (if configured). > Warning: Selecting 'N', or static IP address allocation, disables automatic > provisioning of new hosts in System Inventory, requiring the user to manually > provision using the 'system host-add' command. > Dynamic IP address allocation [Y/n]: Y > Management Network Multicast subnet [239.1.1.0/28]: > > Infrastructure Network: > ----------------------- > > The infrastructure network is used for internal communication between platform > components to offload the management network of high bandwidth services. IP > addresses on this network are reachable only within the data center. > > If a separate infrastructure interface is not configured the management network > will be used. > > It is NOT recommended to configure infrastructure network for All-in- > one duplex-direct. > Configure an infrastructure interface [y/N]: N > > External OAM Network: > --------------------- > > The external OAM network is used for management of the cloud. It also provides > access to the platform APIs. IP addresses on this network are reachable outside > the data center. > > An external OAM bond interface provides redundant connections for the OAM > network. > > External OAM interface link aggregation [y/N]: y > External OAM interface [bond1]: > Configure an external OAM VLAN [y/N]: > External OAM interface MTU [1500]: > > Specify one of the bonding policies. Possible values are: > 1) Active-backup policy > 2) Balanced XOR policy > 3) 802.3ad (LACP) policy > > External OAM interface bonding policy [active-backup]: > A maximum of 2 physical interfaces can be attached to the external OAM > interface. > > First external OAM interface member [enp3s0f0]: enp7s16f1 > Second external oam interface member []: enp7s16f7 > External OAM subnet [10.10.10.0/24]: 10.62.150.0/24 > External OAM gateway address [10.62.150.1]: > External OAM floating address [10.62.150.2]: > External OAM address for first controller node [10.62.150.3]: 10.62.150.210 > External OAM address for second controller node [10.62.150.211]: > > Cloud Authentication: > ------------------------------- > > Configure a password for the Cloud admin user The Password must have a minimum > length of 7 character, and conform to password complexity rules > Create admin user password: > Repeat admin user password: > > > > The following configuration will be applied: > > System Configuration > -------------------- > Time Zone: Europe/Berlin > System mode: duplex-direct > > PXEBoot Network Configuration > ----------------------------- > Separate PXEBoot network not configured > PXEBoot Controller floating hostname: pxecontroller > > Management Network Configuration > -------------------------------- > Management interface name: bond0 > Management interface: bond0 > Management interface MTU: 1500 > Management ae member 0: enp8s20f4 > Management ae member 1: enp8s20f5 > Management ae policy : 802.3ad > Management subnet: 172.27.1.0/24 > Controller floating address: 172.27.1.2 > Controller 0 address: 172.27.1.3 > Controller 1 address: 172.27.1.4 > NFS Management Address 1: 172.27.1.5 > NFS Management Address 2: 172.27.1.6 > Controller floating hostname: controller > Controller hostname prefix: controller- > OAM Controller floating hostname: oamcontroller > Dynamic IP address allocation is selected > Management multicast subnet: 239.1.1.0/28 > > Infrastructure Network Configuration > ------------------------------------ > Infrastructure interface not configured > > External OAM Network Configuration > ---------------------------------- > External OAM interface name: bond1 > External OAM interface: bond1 > External OAM interface MTU: 1500 > External OAM ae member 0: enp7s16f1 > External OAM ae member 1: enp7s16f7 > External OAM ae policy : active-backup > External OAM subnet: 10.62.150.0/24 > External OAM gateway address: 10.62.150.1 > External OAM floating address: 10.62.150.2 > External OAM 0 address: 10.62.150.210 > External OAM 1 address: 10.62.150.211 > > Apply the above configuration? [y/n]: y > > Applying configuration (this will take several minutes): > > 01/08: Creating bootstrap configuration ... DONE > 02/08: Applying bootstrap manifest ... DONE > 03/08: Persisting local configuration ... DONE > 04/08: Populating initial system inventory ... DONE > 05/08: Creating system configuration ... sysinv 2019-02-01 15:20:44.053 25508 CRITICAL sysinv [-] 24 > 2019-02-01 15:20:44.053 25508 TRACE sysinv Traceback (most recent call last): > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/bin/sysinv-puppet", line 10, in <module> > 2019-02-01 15:20:44.053 25508 TRACE sysinv sys.exit(main()) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/cmd/puppet.py", line 75, in main > 2019-02-01 15:20:44.053 25508 TRACE sysinv CONF.action.func(CONF.action.path, CONF.action.hostname) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/cmd/puppet.py", line 47, in create_host_config_action > 2019-02-01 15:20:44.053 25508 TRACE sysinv operator.update_host_config(host) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 30, in _wrapper > 2019-02-01 15:20:44.053 25508 TRACE sysinv func(self, *args, **kwargs) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 145, in update_host_config > 2019-02-01 15:20:44.053 25508 TRACE sysinv config.update(puppet_plugin.obj.get_host_config(host)) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 99, in get_host_config > 2019-02-01 15:20:44.053 25508 TRACE sysinv generate_interface_configs(context, config) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 1029, in generate_interface_configs > 2019-02-01 15:20:44.053 25508 TRACE sysinv generate_network_config(context, config, iface) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 924, in generate_network_config > 2019-02-01 15:20:44.053 25508 TRACE sysinv network_config = get_interface_network_config(context, iface) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 896, in get_interface_network_config > 2019-02-01 15:20:44.053 25508 TRACE sysinv os_ifname = get_interface_os_ifname(context, iface) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 507, in get_interface_os_ifname > 2019-02-01 15:20:44.053 25508 TRACE sysinv os_ifname = get_interface_port_name(context, iface) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 472, in get_interface_port_name > 2019-02-01 15:20:44.053 25508 TRACE sysinv port = get_interface_port(context, iface) > 2019-02-01 15:20:44.053 25508 TRACE sysinv File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 464, in get_interface_port > 2019-02-01 15:20:44.053 25508 TRACE sysinv return context['ports'][iface['id']] > 2019-02-01 15:20:44.053 25508 TRACE sysinv KeyError: 24 > 2019-02-01 15:20:44.053 25508 TRACE sysinv > Failed to update puppet hiera host config > > Configuration failed: Failed to update hiera configuration > localhost:~# > > /tmp/apply_manifest.log: > ======================== > > cp: cannot stat ‘/tmp/hieradata/172.27.1.3.yaml’: No such file or directory > cp: cannot stat ‘/tmp/hieradata/system.yaml’: No such file or directory > cp: cannot stat ‘/tmp/hieradata/secure_system.yaml’: No such file or directory > Applying puppet bootstrap manifest... > [DONE] > > > ls /tmp/puppet/hieradata/ > ========================= > global.yaml personality.yaml secure_static.yaml static.yaml > localhost:~# > > > >
1 0
0 0
[Starlingx-discuss] [build-report] STX_DL_container_setup - Build # 149 - Failure!
by build.starlingx@gmail.com 01 Feb '19

01 Feb '19
Project: STX_DL_container_setup Build #: 149 Status: Failure Timestamp: 20190201T060034Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190201T06… -------------------------------------------------------------------------------- Parameters MY_WORKSPACE: /localdisk/loadbuild/jenkins/master/20190201T060000Z DOCKER_DL_ID: jenkins-master-20190201T060000Z-downloader PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190201T06… DOCKER_DL_TAG: master-20190201T060000Z-downloader-image PUBLISH_LOGS_BASE: /export/mirror/starlingx/master/centos/20190201T060000Z/logs MY_REPO_ROOT: /localdisk/designer/jenkins/master
2 1
0 0
[Starlingx-discuss] [build-report] STX_retag_docker_images - Build # 1 - Failure!
by build.starlingx@gmail.com 01 Feb '19

01 Feb '19
Project: STX_retag_docker_images Build #: 1 Status: Failure Timestamp: 20190131T214007Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/feature/stein/centos/2019… -------------------------------------------------------------------------------- Parameters HOST_PORT: 80 OLD_LATEST_PREFIX: dev MY_WORKSPACE: /localdisk/loadbuild/jenkins/f-stein/20190131T182613Z OS: centos MY_REPO: /localdisk/designer/jenkins/f-stein/cgcs-root BASE_VERSION: f-stein-20190131T182613Z PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/feature/stein/centos/2019… REGISTRY_USERID: slittlewrs HOST: build.starlingx.cengn.ca LATEST_PREFIX: f-stein PUBLISH_LOGS_BASE: /export/mirror/starlingx/feature/stein/centos/20190131T182613Z/logs RETAG_IMAGE_LIST: stx-libvirt FLOCK_VERSION: f-stein-centos-master-20190131T182613Z PREFIX: f-stein OPENSTACK_RELEASE: master TIMESTAMP: 20190131T182613Z REGISTRY_ORG: starlingx OLD_OPENSTACK_RELEASE: pike PUBLISH_OUTPUTS_BASE: /export/mirror/starlingx/feature/stein/centos/20190131T182613Z/outputs REGISTRY: docker.io
2 1
0 0
[Starlingx-discuss] [build-report] STX_build_stein_master - Build # 38 - Failure!
by build.starlingx@gmail.com 01 Feb '19

01 Feb '19
Project: STX_build_stein_master Build #: 38 Status: Failure Timestamp: 20190201T090000Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/feature/stein/centos/2019… -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS: false
1 0
0 0
[Starlingx-discuss] [build-report] STX_DL_container_setup - Build # 150 - Still Failing!
by build.starlingx@gmail.com 01 Feb '19

01 Feb '19
Project: STX_DL_container_setup Build #: 150 Status: Still Failing Timestamp: 20190201T090033Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/feature/stein/centos/2019… -------------------------------------------------------------------------------- Parameters MY_WORKSPACE: /localdisk/loadbuild/jenkins/f-stein/20190201T090000Z DOCKER_DL_ID: jenkins-f-stein-20190201T090000Z-downloader PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/feature/stein/centos/2019… DOCKER_DL_TAG: f-stein-20190201T090000Z-downloader-image PUBLISH_LOGS_BASE: /export/mirror/starlingx/feature/stein/centos/20190201T090000Z/logs MY_REPO_ROOT: /localdisk/designer/jenkins/f-stein
1 0
0 0
[Starlingx-discuss] [build-report] STX_build_master_pike - Build # 124 - Failure!
by build.starlingx@gmail.com 31 Jan '19

31 Jan '19
Project: STX_build_master_pike Build #: 124 Status: Failure Timestamp: 20190201T060000Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190201T06… -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS: false
1 0
0 0
  • ← Newer
  • 1
  • ...
  • 19
  • 20
  • 21
  • 22
  • Older →

HyperKitty Powered by HyperKitty version 1.3.12.