Magento2 - Speed up DI with var_export + include instead of serialize

Poking around in Magento2 I stumbled upon Magento\Framework\ObjectManager\DefinitionFactory which takes care of how the cached DI (Dependency Injections) definitions are loaded. I’ve created a Pull Request but never submitted it.

TL;DR: Use igbinary

Reviewing DefinitionFactory I saw that it only implements serialize and igbinary. My thoughts are: Hey there are faster methods like direct PHP code cached via OPcache! So I’ve implement the var_export and include feature.

Adding the feature

Var_export() generates a PHP parseable array which can be loaded via include() if you write the output of var_export() into a file.

Implementation details of the include() code can be seen here: lib/internal/Magento/Framework/ObjectManager/DefinitionFactory.php.

The CLI script dev/tools/Magento/Tools/Di/compiler.php manages the traversing through all the di.xml files and other PHP files to generate a PHP array with all the dependencies. The var_export() method was quickly implemented with the new class dev/tools/Magento/Tools/Di/Definition/Serializer/VarExport.php which handles the generation of the PHP code.

The whole commit with all changes has been put here https://github.com/SchumacherFM/magento2/commit/09ee71bd6bcffbd30be4d544ffd26ac29e3cc9b1.

Activating the feature

Run the compiler with the new argument:

$ php dev/tools/Magento/Tools/Di/compiler.php --serializer=varexport

That script runs around ~1min with the default Magento2 installation. The new files (definitions.php, plugins.php and relations.php) will be stored in var/di/. Magento2 autodetects the files and loads them immediately.

Add to your app/etc/config.php the entry definition -> format and then the appropriate entry of either igbinary, serialize or varexport.

<?php
return array(
    'definition' => [
        'format' => 'varexport'
    ],
    ...

Now the average speed of your store has been increased. Really?

After everything was setup I’ve loaded the admin/catalog/product/index page a couple of times with the different serializers but the effects are marginal compared to each other.

Searching for other blog posts which covers the topic var_export vs serialize I must write a speed test to finally figure out which serializer provides best performance.

Speed test

Gist of the speed testing script:

I ran this script in the browser instead of on the command line.

Apache and PHP version:

Server version: Apache/2.4.9 (Unix) (bundled with OSX 10.10)

PHP 5.5.20 (cli) (built: Dec 24 2014 11:56:08)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies
    with Xdebug v2.2.5, Copyright (c) 2002-2014, by Derick Rethans

Settings of OpCache:

zend_extension=/usr/local/php5/lib/php/extensions/no-debug-non-zts-20121212/opcache.so
[opcache]
opcache.enable=1
opcache.enable_cli=1
opcache.max_accelerated_files=12000
opcache.memory_consumption=256
opcache.interned_strings_buffer=16
opcache.save_comments=1
opcache.load_comments=1

; dev settings
opcache.validate_timestamps=1
opcache.revalidate_freq=0

Settings of XDebug:

zend_extension=/usr/local/php5/lib/php/extensions/no-debug-non-zts-20121212/xdebug.so
[xdebug]
xdebug.remote_enable=on
xdebug.default_enable=on
xdebug.remote_autostart=off
xdebug.remote_port=9000
xdebug.remote_host=localhost
xdebug.profiler_enable=0
xdebug.profiler_enable_trigger=1
xdebug.profiler_output_name=xdebug-profile-cachegrind.out-%H-%R
xdebug.var_display_max_children = 128
xdebug.var_display_max_data = 2048
xdebug.var_display_max_depth = 128
xdebug.max_nesting_level=200

Results

var_export: ~9 seconds

-rw-r--r--   1 user  _www  3275671 Dec 28 18:02 definitions.php
-rw-r--r--   1 user  _www    16162 Dec 28 18:02 plugins.php
-rw-r--r--   1 user  _www   986948 Dec 28 18:02 relations.php

serialize: ~5 seconds

-rw-r--r--   1 user  _www  2142821 Dec 28 17:59 definitions.php
-rw-r--r--   1 user  _www    13624 Dec 28 17:59 plugins.php
-rw-r--r--   1 user  _www   842291 Dec 28 17:59 relations.php

igbinary: ~4.8 seconds

-rw-r--r--   1 user  _www  1552443 Dec 28 17:56 definitions.php
-rw-r--r--   1 user  _www    10261 Dec 28 17:56 plugins.php
-rw-r--r--   1 user  _www   426936 Dec 28 17:56 relations.php

The file size decreases also and remember that all .php files are stored within the Opcache.

Rants about the unit tests

Running the unit tests:

cd ~/Sites/magento2/site/dev/tests/unit
$ ~/Sites/magento2/site/vendor/bin/phpunit -v --filter DefinitionFactoryTest
$ ~/Sites/magento2/site/vendor/bin/phpunit -v --filter BinaryTest
$ ~/Sites/magento2/site/vendor/bin/phpunit -v --filter SerializedTest
$ ~/Sites/magento2/site/vendor/bin/phpunit -v --filter ObjectManagerTest
$ ~/Sites/magento2/site/vendor/bin/phpunit -v --filter DefinitionFactoryTest
$ ~/Sites/magento2/site/vendor/bin/phpunit -v --filter VarExportTest

Each tests runs around: Time: 23.49 seconds, Memory: 382.75Mb which is pretty annoying to wait so long. Running tests in GoLang takes only milliseconds. Go is for me the first language where I enjoy writing tests, not only due to its speed also due to its idiomatic way.

Optimal settings for Magento2

Your app/etc/config.php should look like:

<?php
return array(
    'definition' => [
        'format' => 'igbinary'
    ],
    ...

Run the compiler:

$ dev/tools/Magento/Tools/Di/compiler.php --serializer=igbinary

Add to your php.ini:

; Use igbinary as session serializer
session.serialize_handler=igbinary

See the How to use section on GitHub.

Don’t freak out when you get this error in your store after reloading PHP:

Warning: igbinary_unserialize_header: unsupported version: 1601398131, should be 1 or 2 in ~/Sites/magento2/site/lib/internal/Magento/Framework/Session/SessionManager.php on line 166

Clear your session cache and reload.

My final pull request

The final PR provides only a change in naming to stay consistent:

Usage: dev/tools/Magento/Tools/Di/compiler.php [ options ]
--serializer <word>           serializer function that should be used (serialize|binary) default = serialize
--verbose|-v                  output report after tool run
--extra-classes-file <string> path to file with extra proxies and factories to generate
--generation <string>         absolute path to generated classes, <magento_root>/var/generation by default
--di <string>                 absolute path to DI definitions directory, <magento_root>/var/di by default
Please, use quotes(") for wrapping strings.

Instead of binary the wording should go to igbinary as this name is also implemented in DefinitionFactory.php line 59.

Related posts

comments powered by Disqus