Transcription

www.phparch.comMarch 2020Volume 19 - Issue 3How Magento isEvolvingThe State of MagentoAlternative Checkout FlowFrom Monolith to ServiceIsolated ArchitectureMagento Inventory andIn-Store Pickup:The Biggest Community ContributionAsynchronous MagentoALSO INSIDEEducation Station:PHP DevelopmentEnvironmentsCommunity Corner:AustinPHPHistory and Computing:My NSA Mug ShotSecurity Corner:Mutual TLSThe Workshop:Managing LAMP withVirtualminfinally{}:Enterprise PHP

Grow your business.Expand your client network.The Magento Solution Partner Program and CommunityInsider Program are now part of the Adobe SolutionPartner Program. Are you ready to help your customersbuild incredible experiences on every screen?Explore the market-leading ecosystem of Adobe experts,learning resources, exclusive benefits, and much more.Join the Adobe SPP today for free atsolutionpartners.adobe.com

Our 15th annual PHP conference,focusing on Tech Leadership.Creator of LaravelTereza NemessanyiKeynoteSpeakersTaylor OtwellCxO & TED SpeakerMay 18-21, 2020Nashville, TN — Oprylandtek.phparch.com

FEATUREAsynchronous MagentoOleksandr LyzunOver the years, Magento architecture has become more sophisticated and complex becausemerchants’ requirements are becoming more complex. This complexity leads to an increase ofapplication infrastructure, release cycles, and the number of features that have to be maintained.For Magento’s first decade, it was a monolith application which nicely covered mid-sized merchantneeds. But as the system grows, the more difficult it becomes to maintain this monolith architecture.Figure 110 \ March 2020 \ www.phparch.comThe final picture with asynchronouscommunication looks like Figure 2.Service decomposition is onlyone example where asynchronouscommunication can be used, but thisis currently the most common example.Now, let’s have a look at the workMagento has already made in:plemay become quite long and lead topainful performance issues. Because ofthe high risk for potential process locks,such a system is tough to maintain andscale. For example, on complex systems,after placing a new order, the systemmay have to execute a long series ofpost-actions, as seen in Figure 1.The potential user experience impactis clear: Processing all of these actionstakes time and depends on all operationscompleting without error—problems,in either case, would be a conversionkiller. Another common considerationis third-party integrations, which maynot be available or are having performance issues. This could mean that thepurchase process is entirely blockedor lasts so long that the customer mayagain cancel their purchase.Let’s proceed with the same example,but this time asynchronously. We cancharge the customer right away, placethe order, and process all required operations to complete the order by usingbackground processes. In case of aproblem or delay with an operation, wecan report this information to the useror retry an operation in the background.The only inconvenience we have left inthis case is failed operations, whichrequire customer input, but this canbe solved with a clearly implementederror-handling business process.SamBeginning in mid-2018, the Magentoarchitecture team began work toachieve a full-service decompositionof the Magento core. The idea is to splitthe Magento framework into separateservices, where each service is fullyisolated and independent and deployable as an independent application.One critical idea with this approach isthe communication between services.Services must be able to communicate with other services, or with themonolith seamlessly. Following one ofthe guidelines of service decomposition, communication between servicesmust be efficient and able to processasynchronously, if possible. This articleexplains why and how asynchronouscommunication was chosen and whichmechanisms Magento uses to achievethis.Let’s have a look at how communication between services can beprocessed. It can be done in either asynchronous or asynchronous manner.Synchronous communication meansall requests have to be processed in thesame order that they were sent, and thesender system must wait for a responsefrom the receiver. This also means theconnection is kept open between thetwo services. If we are talking about acomplex system with many differentservices, the sequence of the operations The direction of asynchronouscommunication Which use cases are already covered Which user cases we are planning tointegrateEvent-Driven Architecture,CQRS, And Event SourcingAsynchronous Magento requiresimplementing a few well-knownarchitectural paradigms. The first isevent-driven architecture1 (EDA). One1 event-driven architecture:https://phpa.me/wikip-edaFigure 2

Asynchronous Magentoa change to the state of a system. Werecord that state change as an event,and we can confidently rebuild thesystem state by reprocessing the eventsat any time in the future.The main ideas around EventSourcing are:2 Command Query Responsibility Segregation: https://phpa.me/mfowler-cqrsreleases, Magento has tried to changethese requests, but could not whollyrewrite all of them. The main reasonis that such changes are not backwardcompatible and may break third-partyintegrations. At the same time, if wefollow CQRS and let go of the RESTfulapproach, such requests have to be eliminated. If you dive deeply into Magentooperations, you can see it is not alwaysnecessary to receive a response immediately after sending the request.Magento already has all the necessary software to be able to integrateCQRS as a standard. The MagentoMessage Queue Framework gives us thebeginnings of an Event Sourcing implementation. All asynchronous operationlogs are currently stored in the Magentodatabase, where you can see all the operations, request, and response messages.But it’s just a first step in the integrationof Event Sourcing.ple There are no limitations on thenumber of events that can happenwith an object. All commands are immutable; thestate of events can never change aftercreation.With Event Sourcing, you alwaysknow what happened with everyobject since its creation. As an example,imagine a product’s stock value, represented with one cell in a database. Eachtime a new order is placed, the stock ischanged based on third-party, or theadmin adjusts the quantity manually,the value of this cell changes withoutrecord. With Event Sourcing, youalways see when and what happenedwith the stock value.CQRS and Event Sourcing areusually tightly connected. And, as weare talking about full-service decomposition in Magento, Event Sourcingcurrently looks like the correct path.Magento’s functionality requires alarge number of processes and objects,and currently, there is a limited transactional record of the values whichchange and the processes involved.Event Sourcing, correctly implemented,provides a complete record of the what/why/how/when of changes, which will:Samof the main advantages of an event-oriented approach is that we don’t needto change the currently existing implementation (module) when we want toadd or change the functionality of thesystem. Implemented properly, developers can add and remove functionalitywithout affecting the stability of thesystem.The second approach is CommandQuery Responsibility Segregation2(CQRS), which involves separatingdatabase writes and reads. In Magento,this means commands are sent toMagento (writes), but the result of thesecommands never contain data from theobjects being communicated with. Onthe other side, we have queries whichreceive information from persistentstorage. In this approach, asynchronous communication introduces thepossibility of inconsistency, as theexecution order may be different thanin a serial execution flow, or in the caseof processing delays, depending oncurrent system loads.Service-orientedarchitecturesmay implement different methods ofcommunication between services. Asan example, let’s have a look at theenterprise service bus (ESB) architectural approach. The ESB provides acentralized messaging between services.The idea is that all service communication goes through a single central point.An ESB implementation, in combination with message queueing, may bequite powerful for implementing aservice-oriented architecture. But thisalso means all services have a strictdependency on the ESB implementationand cannot communicate directly witheach other. As all requests are routedthrough the ESB, this leads to slowercommunication between services.Another disadvantage is that the ESB isalways a single point of failure. A failureof the ESB causes failures across theapplication. This approach brings additional complexity to the architectureand reduces scalability.Event Sourcing is the approach ofstoring the data whenever we make give us control over all systemprocesses, facilitate their planning and development, and allow us to monitor and track thestate of all objects and persist eachwrite operation.Magento Message QueueFrameworkThe main idea of message queues isto provide asynchronous communication between the sender and receiver,without having them communicatedirectly. The sender places the messageinto the queue, and the receiver getsthe message only when the consumerprocesses it.Magento Commerce 2.1 and MagentoOpen Source since version 2.3 providethe Message Queue Framework (MQF).According to Figure 3:Figure 3Challenges to ChangeUntil now, Magento has mostly useda read-write approach. If you madea POST request, you directly receivedinformation about that object. Evenwith this behavior, you can’t say theMagento API was RESTful. In recentwww.phparch.com \ March 2020 \ 11

Asynchronous MagentoUntil recently, the most common usefor MQF comes from shared catalogs inB2B. It is possible to set custom pricesListing 1Sam1. ?php2. 'queue' [3.'amqp' [4.'host' 'rabbitmq.example.com',5.'port' '11213',6.'user' 'magento',7.'password' 'magento',8.'virtualhost' '/',9.'ssl' 'true',10.'ssl options' [11.'cafile' '/etc/pki/tls/certs/DigiCertCA.crt',12.'certfile' 'keyfile' ,15.],16. ],Listing 21. ?xml version "1.0"? 2. config xmlns:xsi oNamespaceSchemaLocation cation.xsd" 5. topic name "synchronous.rpc.test"6.request "string"7.response "string" 8. handler name "processRpcRequest"9.type "Magento\Test\Model\Handler"10.method "process"/ 11. /topic 12. /config 12 \ March 2020 \ www.phparch.comHow to Use MQFThe out-of-box AMQP implementation for Magento is RabbitMQ3. If youread Magento DevDocs:“RabbitMQ is an open-sourcemessage broker that offers a reliable,highly available, scalable, andportable messaging system. Messagequeues provide an asynchronouscommunication mechanism inwhich the sender and the receiver ofa message do not contact each other.Nor do they need to communicatewith the message queue at thesame time. When a sender places amessage onto a queue, it is storeduntil the recipient receives them.”eCurrent Use of MQFfor each customer or to give customerpermissions only for specific categories.Since such operations can take timeand require many system resources,they are performed asynchronously.The next big step forward for MQFuse was a community project whichprovided Asynchronous and Bulk APIsfor Magento. This API is based on amessage queue implementation andnicely demonstrates this new approachto queues. Currently, the MQF is usedin the development of new Magentomodules, as a way for different modulesto communicate with each other. Agood example is the Magento Multisource Inventory module (MSI), whichuses asynchronous communication toassign, unassign, or transfer stock.pl Publisher sends messages to theexchange. Exchange receives messages andsends them to the appropriate queue.Magento currently uses "topicexchanges", where "topic" is a routing key for the message. The topic isa string key consisting of dot-separated parts, which also can containwildcard signs: * for replacing oneof the topic parts, or # to replace allparts after the current. The queue receives and storesmessages. Consumer picks up messages fromthe queue and executes them.RabbitMQ has to be installed andrunning, and we must configureMagento to communicate withRabbitMQ. This can be done by addinga configuration block for the queue toapp/code/env.php shown in Listing 1.To integrate message queueing foryour use cases, you must create fourdifferent configuration files inside ofyour module. Here’s a short overview ofeach of them.communication.xml:Thisfile(Listing 2) contains information aboutmessages the system executes, definedby “topics” and “handlers,” which areresponsible for message processing.Developers have to define a topic namefor the message and define a handler toprocess this message.queue topology.xml:Definesthe routing rules. In this file, definethe destination queue for topics andtransfer some custom arguments thatare passed to the message broker forprocessing (Listing 3 on the next page).queue publisher.xml:Definesthe exchange where the specific topicis published. Currently, Magentosupports only db and amqp connections.Be careful here, as only one exchangecan be enabled for one topic, see Listing4 on the next page.3RabbitMQ: https://www.rabbitmq.com

Asynchronous MagentoListing 3work\MessageQueue\PublisherInterfaceinterface: publisher- publish( topic, message);bin/magento queue:consumers:start \CONSUMER NAMETo get a list of consumers, use thefollowing command, which returns alist of all consumers registered in thesystem.bin/magento queue:consumers:listYou can technically run the sameconsumer several times. This leads toa situation where each consumer takesmessages from the queue and processthem simultaneously. From one pointof view, that’s good, because you canprocess multiple messages at the sametime and speed up the whole process.All cases using the MQF are aimed atimplementing asynchronous communication between application parts, whichdoes not affect the performance of theprocesses themselves but improves thescalability of the entire application.However, there are always boundaries to consider.eListing 41. ?xml version "1.0"? 2. config xmlns:xsi oNamespaceSchemaLocation er.xsd" 5. publisher topic "asynchronous.test" 6. connection name "amqp" exchange "magento" disabled "false"/ 7. connection name "db" exchange "exch1" disabled "true"/ 8. /publisher 9. /config Listing 5SamAfter that, your message is publishedin the queue.To execute a Magento queueconsumer, run the command:1. config xmlns:xsi oNamespaceSchemaLocation y.xsd" 4. exchange name "magento-topic-based-exchange1" type "topic"5.connection "db" 6. binding id "topicBasedRouting2" topic "anotherTopic"7.destinationType "queue" destination "topic-queue1" 8. arguments 9. argument name "argument1"10.xsi:type "string" value /argument 11. /arguments 12. /binding 13. /exchange 14. /config plqueue consumer.xml: Defines theconsumer that processes messages fromthe queue (Listing 5).When the MQF was first created, itcontained only the queue.xml configuration file. But working with messagequeues requires flexibility in customization, so the next step was to splitthe configuration such that messagequeueing is easy to configure andeasy to extend. In the case of isolatedservices, one goal is making it possiblefor a Publisher to be deployed andconfigured in one environment andthe consumer on another—with eitherknowing nothing about the other!After creating all required configurations, your message is basically readyto be processed. To send a messageto the queue, you have to execute thepublish() method of \Magento\Frame-1. ?xml version "1.0"? 2. config3.xmlns:xsi oNamespaceSchemaLocation r.xsd" 6. consumer name "basic.consumer"7.queue "basic.consumer.queue"8.handler "LoggerClass::log"/ 9. /config Mysql Deadlocks. In case themessage processing requires a lotof data modifications on persistentstorage, this means the same operation may try to create or modify datain the same tables at the same pointin time. Depending on the tables andsystem architecture, this may leadto an issue where the first messagelocks the table. When another onetries to write data on it, the systemthrows an exception, and theoperation cannot be completed. TheMagento team has eliminated muchof these, but some ultra-high-scalesituations still require investigationand thoughtful design. Performance. Multiple simultaneous message execution requiresmore system resources. For example, creating hundreds of productssimultaneously requires far moreresources than serving hundreds ofcustomer requests.An additional consideration: Allmessages that the system creates andtransfers to the message queue have tobe idempotent. Idempotence is perhapsfamiliar to those of you who haveworked with or built RESTful services.It means the same operation can beexecuted multiple times, but the resultalways stays the same. This propertyfacilitates using multiple queues orswapping out queue implementations.www.phparch.com \ March 2020 \ 13

Asynchronous MagentoRunning Consumers'cron consumers runner' ['cron run' false,'max messages' 20000,'consumers' ['consumer1','consumer2',]]Let’s have a look at the RabbitMQ messages the systemsends to the queue.Packing of messages happens in this ublisher::publish( topicName, data)SaIt method creates envelopes for each message and sendsthem to the queue.RabbitMQ messages have a predefined list of messageproperties, but the current Magento implementation usesonly a couple of them, shown in Listing 6. body is our message body which contains ID, topic name,bulk uuid, and serialized data. delivery mode is a constant and means this message has apersistent delivery mode.Listing 61.2.3.4.5.6.7.8.9.10.Magento provides a REST API giving developers and integrators access to almost all Magento features. It can be usedfor smooth and efficient communication between Magentoand third-party systems. Theoretically, this approach also canbe used for communication between Magento and Magentoservices. However, for this purpose, there are some challenges which you can guess based on the other points. First,all REST requests are synchronous. Doing so leads to theproblem where each system executing an API request mustwait for the response from the system to be sure that theoperation executed successfully. If an operation requires a lotof time to execute, the whole process is slowed down. If wetry to send multiple API requests at the same time, this mayhave a significant impact on system performance. Also, theREST API does not allow us to transfer multiple messages toMagento at the same time.The reason why this happens is that the initial idea was tosupport a single point of customization, and it was assumedthat if a developer needed to handle N entities, they wouldcreate a single message. But in the case of large data operations, developers quickly hit deadlocks or resource limits.These points were the main reason why the communitystarted to develop asynchronous and Bulk APIs for Magento,which has moved Magento APIs to the next level.mIn the “consumers” section, you should define a list ofconsumers you want your cron to execute.RabbitMQ MessagesSynchronous Web APIplCron jobs are the default mechanism to start consumers.In the configuration, it is possible to define the number ofmessages that are processed by a consumer. After reachingthis number, the consumer is terminated. Re-running cronrestarts the consumer.To enable consumers for your system, you have to addfollowing configuration in your /app/etc/env.php file: message id is a unique id for the message.One new feature released in Magento version 2.3.3 is theuse of message “application headers.” Now each messagereceives a new parameter, store id. This parameter notes thecurrent store for which a request was executed. We need thisinformation to apply the message to the correct system scope.When a consumer picks up a message from the queue, itreceives the same information that was transferred from thepublisher. Then based on communication.xml and the messagebody, it can process the correct operation.eIt’s also essential to find the sweet spot of running consumers,with a preference for messages that trigger bulk operations,which partially resolve problems with locking. Time of dayis also vital to consider; for example, it makes sense to runimport operations at night—or whenever customer impact islowest. envelopes[] this- envelopeFactory- create(['body' message,'properties' ['delivery mode' 2,'message id' this- messageIdGenerator- generate( topicName),]]); 14 \ March 2020 \ www.phparch.comAsynchronous/Bulk APIIn discussing asynchronous Magento, we definitely haveto cover the topic of using asynchronous API as the primarycommunication mechanism between services.Asynchronous API is implemented on top of the REST API.When the system receives API requests which have to beexecuted asynchronously, it adds them to the queue. At thesame time, the consumer reads messages from the queue andprocess them.Bulk API is a community-driven implementation on topof asynchronous API. It allows us to send a single requestwith multiple objects in the body. The system splits theseobjects to single messages and executes them asynchronously,providing substantial benefits:1. The response time of all asynchronous bulk requestsare faster than the response time of typical synchronous REST requests, yielding approximately 30% of

plperformance improvements; in theFigure 4case of Bulk API, this change istremendous.2. Bulk API provides the possibilityto send multiple objects in onerequest. Doing so leads to a reduction in the total number of requests,further reducing process overhead.Let’s have a look at some stats—thesetests were made on Magento Cloudinfrastructure with 16 CPUs, 256 GBRAM, and Magento 2.3.2 CommerceEdition installed. In Figure 4, you cansee the performance comparison ofdifferent types of APIs when receivingmessages.We performed tests where we startedto send simultaneous requests toMagento API product-create endpoints,starting from one request and endingwith 150 simultaneous requests. YouFigure 5can see that for synchronous and asynchronous APIs, average response timesfor the maximal amount of requests areclose to 90 and 80 seconds, respectively(each request contains one object). Ifwe have a look at 150 objects in a singlerequest of the BULK API, Magento wascapable of receiving 150 objects withinapproximately 10 seconds for all ofthem—an eight-fold improvement!This performance test offers anotherimportant metric—incidence of errors,sync versus async—shown in Figure 5.Each system has its resource limitations and limitations of server software.The system in this test had a limitationof 134 simultaneous processes running.Starting at 135 requests, the systemstarted to throw 50x errors back withoutprocessing those messages. You cansee how the number of failures growsfrom the number of requests—red andblue lines are the same here. The sameBulk API does not have such problemsbecause it only sends one request independent from thenumbers of objects. The same situation happens with serverresources because, with high numbers of requests, CPU andRAM usage is growing, respectively. Definitely, during BulkAPI usage, you have to be aware of some other system settings,like the PHP setting max post size, but those are not dependent on hardware.As I mentioned, the Bulk and asynchronous API implementation works atop the usual REST API. Developers don’t needeAsynchronous MagentoSamto take care of any additional development to make their APIsupport Bulk operation. By default, Magento takes all POST,PUT, or DELETE REST endpoints and makes them available asBulk. In addition, if third-party developers are using PATCHrequests, they are also automatically supported by Bulk API.In the case of asynchronous APIs, everything is quitesimple. To the usual REST URL POST {{URL}}/rest/all/V1/products the async prefix is added: {{URL}}/rest/all/async/V1/products. Then, the system recognizes this request as asynchronous and processes it.www.phparch.com \ March 2020 \ 15

Asynchronous Magento[{"block id": "1"},{"block id": "2"}]1. [2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17. ]{"product": {"sku": "SKU1","name": "Simple product 1","attribute set id": 4}},{"product": {"sku": "SKU2","name": "Simple product 2","attribute set id": 4}},.plrequest have to be transferred as an array (Listing 7).Some API requests also contain parameters in URLs, forexample: DELETE /V1/cmsBlock/:blockId, which expects tohave the exact block ID to be transferred as a URL parameter.As you can understand, in case of multiple object transfer,we had the challenge to create some solution that does notrequire any additional development from the developers’ side,and at the same time, give integrators the flexibility in APIusage. The idea is to automatically convert those URL parameters to the static part of the URL. For Bulk API usage, URLshave to be converted such that “:” has to be replaced with “by”prefix and next letter converted to uppercase. For example, “/V1/cmsBlock/:blockId” has to be changed to “/V1/cmsBlock/byBlockId”. And at the same time, the block id parameter hasto be added to the object body.Listing 7Lis ting 81.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.{SamBy the way, if in the REST API, DELETE requests have nobody, then in the Bulk API request, the body of such requestscontains an array of corresponding object IDs.The response of all async and Bulk requests is always thesame as you can see in Listing 8. It is the UUID of the operation and a list of items (in case of asynchronous API, alwaysone item):The Bulk UUID can be used later to receive operation statusand also to check the response of the operation. The UUIDis always unique and always represents one bulk operation.Request the status of the operation by calling:GET /V1/bulk/:bulkUuid/detailed-statusAnd the answer you receive looks like Listing 9.Asynchronous and Bulk API functionality become available for developers starting in version Magento 2.3. And wewant to thank Magento partners: comwrap GmbH (Frankfurt, Germany) and Balance Internet (Melbourne, Australia)who made this possible.Working with QueuesTo enable the processing of Bulk APIs, you have to specifya consumer to be run. Consumer name for asynchronous andBulk API is “async.operations.all.” This consumer receivesand processes all requests that come to the async/Bulk API,which means you have one master consumer for all APIs.If you want to speed up processing, you can run multipleconsumers. If you are sure your operations are independent16 \ March 2020 \ www.phparch.comeIn the case of Bulk API, the usage is a POST {{URL}}/rest/all/V1/products with an async/bulk prefix: {{URL}}/rest/all/async/bulk/V1/products, and also, items inside of the body of"bulk uuid": "799a59c0-09ca-4d60-b432-2953986c1c38","request items": [{"id": 0,"data hash": null,"status": "accepted"},{"id": 1,"data hash": null,"status": "accepted"}],"errors": false}Listing 91. {2."operations list": [3.{4."id": 4,5."bulk uuid": "c43ed402-3dd3-4100-92e2-dc5852d3009b",6."topic name": face.createaccount.post",7."serialized data": "{\"entity id\":null,\"entity link\":\"\",\"meta anie \"}\"}",8."result serialized data": null,9."status": 3,10."result message": "A customer with the same emailaddress already exists in an associated website.",11."error code": 012.}13. }

Asynchronous Magento exchange name "magento" type "topic" connection "amqp" binding id "async.operations.all"topic "async.#"destinationType "queue"destination "async.operations.all"/ /exchange products processing and another for customers and forwardthe required topics to them. exchange name "magento" type "topic" connection "amqp" binding id ace"topic ace.#"destinationType "queue"destination "async.magento.catalog.api.product" / binding id "async.magento.customer"topic "async.magento.customer.#"destinationType "queue"destination "async.magento.customer"/ /exchange eof each other and you want to make the whole communication process much faster—for example, checkout process andcustomer registration—you can configure the system so eachconsumer processes only its objects.Currently, the system is configured so all topic names thatstart with “async.#” are delivered into “async.operations.all”queue. You can find this definition in app/code/Magento/WebapiAsync/etc/queue topology.xml.Topic names for asynchronous and BULK API areauto-generated based on service contract names defined foreach API request.Let’s have a look Magento/Catalog/etc/webapi.xml:And only one consumer is defined in queue consumer.xml: route url "/V1/products/:sku" method "PUT" service class thod "save" / resources resource ref "Magento Catalog::products" / /resources /route pl consumer name "async.operations.all"queue "async.operations.all"connection "amqp"consumerInstance "/ All topic names are generated as PREFIX service class service method route.method. Prefixes always equalto async. The service class is a class na

Asynchronous Magento Publisher sends messages to the exchange. tion for Magento is RabbitMQExchange receives messages and sends them to the appropriate queue. Magento currently uses "topic exchanges", where "topic" is a rout-ing key for the message. The topic is a string key c