Manuscript DetailsManuscript numberIOT 2018 63TitleThe Internet of Things, Fog and Cloud Continuum: Integration and ChallengesShort titleIoT-Fog-Cloud ContinuumArticle typeReview ArticleAbstractThe Internet of Things needs for computing power and storage are expected to remain on the rise in the next decade.Consequently, the amount of data generated by devices at the edge of the network will also grow. While cloudcomputing has been an established and effective way of acquiring computation and storage as a service to manyapplications, it may not be suitable to handle the myriad of data from IoT devices and fulfill largely heterogeneousapplication requirements. Fog computing has been developed to lie between IoT and the cloud, providing a hierarchyof computing power that can collect, aggregate, and process data from/to IoT devices. Combining fog and cloud mayreduce data transfers and communication bottlenecks to the cloud and also contribute to reduced latencies, as fogcomputing resources exist closer to the edge. This paper examines this IoT-Fog-Cloud ecosystem and provides aliterature review from different facets of it: how it can be organized, how management is being addressed, and howapplications can benefit from it. Lastly, we present challenging issues yet to be addressed in IoT-Fog-Cloudinfrastructures.KeywordsInternet of Things; Fog Computing; Edge Computing; Cloud ComputingCorresponding AuthorLuiz Fernando BittencourtCorresponding Author'sInstitutionUniversity of CampinasOrder of AuthorsLuiz Fernando Bittencourt, Roger Immich, Rizos Sakellariou, Nelson Fonseca,Edmundo Madeira, Marilia Curado, Leandro Villas, Luiz daSilva, Craig Lee,Omer RanaSubmission Files Included in this PDFFile Name [File Type]FogIoT-main.pdf [Manuscript (without Author Details)]To view all the submission files, including those not included in the PDF, click on the manuscript title on your EVISEHomepage, then click 'Download zip file'.

The Internet of Things, Fog and Cloud Continuum:Integration and ChallengesAbstractThe Internet of Things needs for computing power and storage are expectedto remain on the rise in the next decade. Consequently, the amount of datagenerated by devices at the edge of the network will also grow. While cloudcomputing has been an established and effective way of acquiring computationand storage as a service to many applications, it may not be suitable to handlethe myriad of data from IoT devices and fulfill largely heterogeneous applicationrequirements. Fog computing has been developed to lie between IoT and thecloud, providing a hierarchy of computing power that can collect, aggregate,and process data from/to IoT devices. Combining fog and cloud may reducedata transfers and communication bottlenecks to the cloud and also contributeto reduced latencies, as fog computing resources exist closer to the edge. Thispaper examines this IoT-Fog-Cloud ecosystem and provides a literature reviewfrom different facets of it: how it can be organized, how management is being addressed, and how applications can benefit from it. Lastly, we presentchallenging issues yet to be addressed in IoT-Fog-Cloud infrastructures.Keywords: Internet of Things (IoT), Fog Computing, Edge Computing,,Cloud Computing1. IntroductionElectronic devices have been expanding their ability to generate data, resulting in the accumulation of a wide variety of information ranging from measurements of natural phenomena to human-related behavior. With the expected5expansion of the Internet of Things (IoT) [1], it is predicted that in the futurePreprint submitted to Internet of ThingsSeptember 3, 2018

virtually all objects will be in some way connected. Along with the connectionof everything to the Internet comes the need for transferring, storing, and processing unprecedented amounts of data, laying down a path for many years ofresearch related to such IoT requirements.10Cloud computing has evolved and became an easy-to-use platform for applications in general to store and process data. IoT devices frequently rely oncloud computing to store and process data, producing information and knowledge as a result. On the one hand, the wide adoption of cloud computing isa consequence of a fast time-to-market for many types of applications due to15the paradigm’s flexibility and reduced or null initial capital expenditures. Onthe other hand, this same wide adoption has exposed some limitations of theparadigm in fulfilling all requirements of some classes of applications, such asreal-time, low latency, and mobile applications. The centralized cloud data centers are often physically and/or logically distant from the cloud client, implying20communication and data transfers to traverse multiple hops, which introducesdelays and consumes network bandwidth of edge and core networks [2].The widespread adoption of cloud computing, combined with the ever increasing ability of edge devices to run heterogeneous applications that generateand consume all kinds of data from a variety of sources, requires novel dis-25tributed computing infrastructures that can cope with such heterogeneous application requirements. Computing infrastructures that enact applications atedge devices have started to appear in recent years [3, 4], improving aspectssuch as response time and reducing bandwidth use. Combining the ability ofrunning smaller, localized applications at the edge with the high-capacity from30the cloud, fog computing has emerged as an paradigm that can support heterogeneous requirements of small and large applications through multiple layersof a computational infrastructure that combines resources from the edge of thenetwork as well as from the cloud [5].In this paper, we aim at identifying and reviewing the main aspects and35challenges that make the combination of fog computing and cloud computing suitable for all kinds of applications leveraged by the Internet of Things.2

We discuss aspects from the infrastructure (processing, networking, protocols,and infrastructure for 5G support) to applications (smart cities, urban computing, and industry 4.0), passing through the management complexity of the40distributed IoT-fog-cloud system (services, resource allocation and optimization, energy consumption, data management and locality, devices federationand trust, and business and service models).In the next section we introduce concepts and definitions for Internet ofThings (IoT), cloud computing, and fog computing. In Section ? we review the45literature on the aforementioned infrastructure, management, and applicationsaspects. Section ? discusses several challenges related to these aspects, andSection ? presents concluding remarks.2. IoT, Fog, and Cloud: Basic DefinitionsThis section introduces the terminology and concepts related to the three50components of the IoT-Fog-Cloud ecosystem.2.1. Internet of ThingsSeveral predictions about the Internet of Things have been put forward inthe literature. If one consensus exists about IoT, it is about the number ofconnected devices: dozens of billions of “things will be connected in a few years55from now [1, 6]. Such devices can include virtually any object with embeddedmicrocontroller and communication capabilities (e.g., in a generalized manner,a set of sensors and/or actuators).This unprecedented number of devices results in an unprecedented amountof data to be transmitted and processed. More than that, IoT connected de-60vices are highly heterogeneous at many levels: data communication protocols,energy requirements, computing capacity, mobility, and so on. Therefore, IoTdevices management, throughout the data communication and processing stack,becomes intrinsically challenging.Raw data generated by the Internet of Things as a whole may not be directly65useful. Such extraordinarily large data sets require significant processing and3

knowledge extraction capabilities to provide some insightful information. IoTapplications are aimed at realizing this task: transform gathered data into actualinformation knowledge. Although a myriad of new applications is enabled byIoT, this is also a source of increased heterogeneity: different applications also70have different requirements, which should be fulfilled by the computing systemamalgamating IoT devices with their applications. In the next sections, wepresent two computing paradigms that can be utilized together to fulfill theheterogeneous requirements associated with IoT applications: cloud and fogcomputing.752.2. Cloud computingCloud computing has achieved a mature state in the past decade, turninginto a widely adopted computing paradigm for many applications, due to itsdynamic characteristics such as elasticity and pay-per-use. To be able to providethese characteristics, virtualization is one of the management pillars for cloud80providers. Virtual machines and containers allow providers to share slices oftheir computing resources, usually deployed in large data centers, among users,resulting in a logically isolated system for each tenant.On-demand computing is offered by cloud providers based on three canonicalmodels, namely Infrastructure as a Service (IaaS), Platform as a Service (PaaS),85and Software as a Service (SaaS) [7]. IaaS offers computing infrastructure as aservice, where the user can remotely access and manage computing power; PaaSoffers a platform for software development along with the necessary libraries anddatabases to deploy and run applications, and SaaS offers the software itselfrelying on the cloud providers infrastructure to offload computing and/or data.90A variety of cloud service levels has surfaced, resulting in the Everything as aService (XaaS) concept [8].Cloud providers can also be classified according to their deployment model:public, private, hybrid and community clouds. Public clouds are those open tothe public, usually charged on a pay-per-use basis for anyone with an Internet95connection. Private clouds are restricted to a set of predefined users (e.g.,4

from a company or university). Hybrid clouds are a composition of public andprivate cloud resources, often composed to fulfill the dynamic demand and avoidupfront investment for peak demand [9]. The community clouds [10], resemblingthe virtual organizations from grid computing[11], are a composition of private100clouds in order to share resources.Cloud services are offered based on a Service Level Agreement (SLA), whichestablishes what is offered and how the user should be charged to use the cloudservice. Common examples are pay-per-use models where charging takes placeby time unit (e.g., virtual machines in a per hour basis), by amount of data105(e.g., data transfers off the provider or amount of data stored), or by numberof requests (e.g., the number of times a specific function/method was called inthe programming model of a SaaS provider).The above characteristics result in properties that make the cloud attractivefor clients, as for example on-demand provisioning/deprovisioning, elasticity,110ubiquitous access, lower upfront investments with reduced capital expendituresin exchange for operational expenditures, and faster time to market. Throughout this paper, we discuss how clouds can fulfill part of the application requirements within the IoT landscape. We also discuss how fog computing, defined inthe next section, can be combined with the cloud to provide an infrastructure115that fulfills a wide range of requirements for IoT applications.2.3. Fog ComputingComputing capacity at the edge increased with the hardware evolution ofpersonal devices. The combination of higher computing capacity with the concomitant evolution of edge networks leveraged distributed computing paradigms120that propose the utilization of edge devices to run applications and store data.The hardware evolution also allowed devices to shrink in size, resulting in mobiledevices that have enough computing and battery capacity to run applicationswith reasonable complexity and quality of service (QoS).The aggregation of edge devices into a distributed system infrastructure has125different names in the literature, also showing different characteristics and fo5

cuses. For example, the European Telecommunications Standards Institute (ETSI)recently changed the name of Mobile Edge Computing to Multi-access edge computing, while keeping the same MEC acronym [12]. This change is an effort toprovide a more flexible framework which goes beyond the new cellular operator’s130requirements. The primary goal is still the same, that is, to provide cloud-likefeatures close to the subscribers at the edge of the network, however, it nowincludes all other wired and wireless communication technologies. Because ofthat, a broad range of new designs can be implemented at, for example, IoTand Vehicle-to-everything (V2X) networks.135Fog computing brings together the edge devices and the cloud, as well asintroduces a hierarchy of computing capacity (fog nodes, cloudlets or micro datacenters) between the edge and the cloud [13]. This capacity can be scatteredat access points, routing devices in the network, the network core, and so on.It is expected that the higher in the network hierarchy a fog node (cloudlet or140micro data center) is, the larger its computing capacity since it should providecapacity for a larger set of users downwards the hierarchy. Moreover, the lowerin the hierarchy a device is, the closer to the edge it is, thus presenting lowercommunication delays to edge devices (e.g., end-user devices, IoT sensors andactuators, vehicles, drones, and so on).145The computing hierarchy in the fog infrastructure can offer a wider rangeof service levels, supporting applications that cannot be supported by cloudcomputing alone. A fog infrastructure is able to handle applications with avariety of QoS requirements, as applications can run at a hierarchy level thatprovides adequate processing capacity and meets latency requirements. Another150consequence of the use of processing closer to the edge is to reduce (aggregate)bandwidth use in the network along the path between edge and cloud.3. Literature reviewIn this section we discuss and review three different facets of the IoT-FogCloud hierarchy, as illustrated in Figure 1:6

1551. Infrastructure, where the computing and networking infrastructure ofcloud and fog is defined and discussed, including networking aspects interms of infrastructure connectivity as well as protocols to access the infrastructure, and the usefulness of fog computing in support of 5G;2. Management, where management needs for the IoT-Fog-Cloud infrastructure are discussed and reviewed, including orchestration, resource man-160agement, services management, energy consumption, devices federation,and data locality;3. Applications, where three different kinds of applications are considered,namely urban computing, mobile applications, and the Industrial IoT, discussing how they can benefit from fog computing.165Resource allocation and scheduling, services management,energy consumption, data locality, federation, orchestrationCloud, Fog, Networking, 5G infrastructuresCloudFog(layer 2)UrbanMobileIndustryIoT applicationsFigure 1: Illustrative overview, within the IoT-Fog-Cloud infrastructure, of topics covered inthis paper.7

3.1. Infrastructure3.1.1. Cloud and FogThe infrastructure discussed in this paper is a composition of fog and cloudto support IoT applications, therefore constituting a three-tiered infrastructure.170While IoT devices are concentrated at the edge of the network, fog devices aredistributed from the IoT device’s access point through the network core. Thecloud is further away from the IoT sensors/actuators, requiring requests fromthe edge to traverse the public Internet to access cloud computing resources.As the fog infrastructure can be itself composed of different levels [13], the175mid-layer of the IoT-Fog-Cloud infrastructure can offer a variety of levels ofquality of service [14]. Applications that have different requirements can bedeployed and run on any device in this infrastructure composition, dependingon their requirements. Moreover, application components can be distributedamong devices at different levels of the fog depending on the application needs180(e.g., latency, computing capacity, data locality).Cloud computing services are based on centralized data centers, where computing capacity is offered over the virtualization of computing clusters deployedin buildings specially designed to host them. Hosts in the data center are oftenconnected through Ethernet; different topologies for this interconnect are avail-185able in the literature [15, 16]. Cloud users are usually not aware or concernedabout details of the data center network topology, even though this can havean impact on the application’s behavior. As the infrastructure managementand control are provider’s duties, how this infrastructure impacts applicationshould be reflected in the Service Level Agreement (SLA) established between190providers and users.Cloud data centers are large facilities deployed in a limited number of locations due to special infrastructure requirements, such as space, power, andcooling, as well as due to the need for qualified workforce and the associatedmanagement costs. On the other hand, cloud users are scattered worldwide,195and consequently many users are not geographically close to cloud data centers8

of their preferred cloud provider. In the same way, IoT devices are scatteredand may also be distant from the cloud; thus, the fog computing infrastructurecan be closer to those devices to bring computing capacity with lower responsetime.200The fog infrastructure can be organized in a hierarchy among the edge devices (IoT, mobile smart devices, etc) and the cloud data centers. The distribution (e.g., density or number of levels) of this hierarchy can vary from placeto place, but the first level is expected to be located one hop away from theedge (user or device): at the access point (e.g., WiFi or cell phone antennas)205or immediately above it. This would be the first (closest) offloading option fordevices at the edge, providing lower latencies even though with limited computing capacity. This single level of processing can be combined with the cloudto provide the necessary computing power for applications with heterogeneousrequirements [17], but other fog levels may be added to enhance computing210capacity closer to the edge and allow data processing/transit between devicesconnected to different or distant access points. The multi-tier deployment offog nodes may depend on the use case. The number of tiers of a fog system isdetermined by different factors, such as the characteristics of the workload tobe processed, available capacities of processing nodes, number of sensors and215actuators, and latency requirements. The workload on fog nodes is generallyrelated to the processing of latency-constrained jobs. The cloud can processheavy workloads and perform long-term storage of data.It is common to designate the aforementioned hierarchy of computing capacity as fog nodes [18], cloudlets [19] or micro data centers [20, 21]. Conceptually,220the higher in the hierarchy a cloudlet is, the larger its processing/storage capacity is, since it is expected to support more devices in the tree downwards theedge. On the other hand, cloudlets that are higher in the hierarchy are also expected to present longer network delays to the edge. Therefore, the hierarchicalcomposition of micro data centers (or cloudlets) along with the cloud provides225a range of computing capacity at different geographical (and logical) distancesto the IoT devices at the edge.9

3.1.2. Networking and Fog HierarchyThe connectivity between several tiers in the fog/cloud hierarchy can bepossible thanks to several network technologies, including wired and wireless230ones. Figure 2 illustrates this fog/cloud hierarchy connectivity.CloudFogLayer N FogLayer 2FogLayer 1Fog-enabledApplicationsFigure 2: IoT-Fog-Cloud connectivity.The functional unit of a fog computing system, i.e., the fog node, can beeither a logical or a physical entity, embedding computing, storage, and alsonetworking capabilities. Diverse devices can act as fog nodes, including networking devices: routers, switches, wireless access points. Along with other235fog nodes (e.g., video surveillance cameras and traditional servers) networkingdevices can also enable the processing of tasks closer to data sources, providingincreased performance and security critical to health, military, or emergencyapplications.Data in IoT-Fog-Cloud infrastructures must traverse one or more tiers, con-240nected by a fog network. The decision on how to connect different nodes dependson a specific technological scenario. For example, a fog node used to processraw data from sensors will typically employ wireless connections, while a fog ina factory employed to monitor manufacturing processes is likely to use wired10

connections.245The connection to the cloud is made by the Internet. It typically employsoptical links belonging to an Internet Service Provider that will connect fognodes to the Internet while interconnection between different fog nodes arelikely to be made by Ethernet-like protocols. Moreover, the network topologyconnecting different fog nodes will depend on how communication channels are250placed a given area, and thus different topologies and network technologies toconnect fog nodes at the same or different layers can co-exist.Wireless links may typically connect IoT devices to the fog since these devicesoften have only wireless interfaces. Wireless connections can also be employedin fog-to-fog or fog-to-cloud interconnections depending on the available infras-255tructure. Cellular technologies (3G, 4G, 5G) are expected to be used in fogcomputing systems. For example, an architecture for fog computing, namedTelcofog, has been designed to integrate fog nodes in 5G networks [22]. In thisarchitecture, a fog node can be created in an edge network and accessed byend-users using 5G connections.260Wireless LAN (WLAN) networks are also useful in the deployment of fognodes. Due to their limited range, they can be used in fog nodes for small buildings or campus [23, 24]. Other wireless technologies for device-to-fog communication include Wireless Personal Area Networks (Bluetooth, ZigBee, Infrared)and Near Field Communication. Moreover, multi-hop wireless transmissions to265route data between sensors/actuators and the fog have also been proposed inthe literature [25].3.1.3. Data collection and Protocols at the EdgeAt the lowest level of an IoT network, it is expected to have hundreds or evenmore elements associated with each fog node. These elements may be sensors,270actuators, or control nodes, which can range from relatively dumb devices, withmajor processing capability constraints, to well-equipped nodes that can easilyhandle the full stack network protocols [26].Different devices may have distinct needs regarding network protocols they11

implement. Because of that, it is not possible to address this issue with an275one-fits-all approach. To make matters worse, scenarios where fog and edgecomputing best fit are those where data needs to be collected, analyzed, andacted upon within a few milliseconds. In the light of the above, the networkprotocols should accommodate the requirements of such communication patterns by providing the flexibility, scalability, and availability needed. This holds280especially true for delay-sensitive applications.Another important aspect in the IoT-Fog-Cloud hierarchy is that the communication technologies are going to be responsible to connect heterogeneousdevices so that they can enable new smart services. The physical objects willbe bridged together at the edge of the network to provide intelligent decision285making capabilities by talking to each other, sharing information, and coordinating decisions without human intervention. It is important to notice that aconsiderable amount of IoT nodes are power-constrained and need to operate innoisy or lossy communication links. Placing an excerpt of the intelligence at thefog level helps to reduce some of these device-related complexity constraints.290The most fitted network connectivity model may widely range depending onthe nodes location, purpose, and specifications [27]. Nodes may be connectedthrough a wired network, for example, on a factory floor or other highly noisyenvironment. This connection is also a good option for static nodes that requirevery high throughput. One example of a communication protocol, in this case,295is the Ethernet [28], with either copper or fiber links. On the other hand,nodes with mobility or deployed at places without a pre-existing infrastructurecan use wireless networks. Examples of communication protocols in the lattercase are Wi-Fi [29], Bluetooth low energy [30], Z-wave [31], IEEE 802.15.4(WirelessHART, ZigBee, ISA100.11a, MiWi) [32], just to name a few.300Another important advance in communications protocol technology is theIPv6 over Low-Power Wireless Area Networks (6LoWPAN) standard [33]. Thistechnology contemplates an adaptation layer between the network and link layers to adjust the size of the packets, which are smaller in the IEEE 802.15.4standard. In addition, it also implements several header compression techniques12

305to handle resource-limited devices. Moreover, the IPv6 allows having a muchlarger address space and also the capability of establishing a direct externalcommunication link between the devices without the need for a coordinator ora gateway to translate the messages [34].Taking everything into consideration, having the sensors, actuators, and310control nodes close together at the fog level brings several advantages. However,several of them may not be capable of communicating directly with a fog node.Because of that, protocol abstraction layers are needed to logically connect allthese elements as well as data collection and aggregation to consolidate the data.The main goal of data collection and aggregation is to use a centralized315approach to gather all the data in an effective manner, which provides severalbenefits [35]. First of all, because IoT elements are close to the edge nodes, thefog can act as the first front of access control and encryption as well as provideisolation and contextual integrity. Besides, it can also protect privacy-sensitivedata [36, 37], by not allowing it to leave the edge. In addition, sensors often320generate a huge volume of raw data in burst mode, i.e., there are peaks of datatransmission that fluctuate over time. By collecting this data and aggregatingthe results it is possible to reduce (if not eliminate) redundancy, which leads toa decrease in network communication cost [38]. This can also have a positiveimpact on the network lifespan, improve the energy consumption [39], prevent325traffic bottlenecks, as well as enhance data accuracy by removing outliers andmisreadings [40].Another advantage of this technique is to provide data locality. This meansthat it is possible to give context to the data and handle it where it makes themost sense. This allows making the decision process more efficient as it helps to330act as soon as the raw data is converted into some meaningful context; furtherinformation can be found in section An Infrastructure to Support 5GCurrent developments in 5G are characterized by the orchestration of network resources to meet a wide range of services, broadly classified into three13

335categories: enhanced mobile broadband (eMBB), ultra-reliable and low-latencycommunications (URLL), and massive machine-type communications (mMTC).While a fog/cloud computing hierarchy plays a role in all three, IoT services aregenerally considered under the mMTC heading. The idea is that a 5G operatorwill be able to slice the network, provisioning each slice to meet the very diverse340needs of the three types of service, in terms of latency, reliability, throughput,scalability, and mobility support. A slice, in this context, can be thought of as avirtual network, whose resources are provisioned for a particular service or classof service and isolated from other slices that share the same physical infrastructure. They are expected to be one of the key resources in 5G networks [41],345by providing a holistic end-to-end virtual network for a given user, so-calledtenants. This means that the physical mobile network will have its resourcespartitioned and customized according to the system needs [42]. The outcome isa more flexible, reliable, scalable, and secure network. Using these technologies,in many situations, the networks will be able to reconfigure slices within seconds350to quickly respond to local demands, such as an unexpected gathering of peopleor to prioritize emergency systems. On the other hand, it is also possible toprogram a long-term lease, for example, to an electrical utility company to accommodate its smart grid components such as meters, sensors, controllers, andother IoT devices. A short-term lease is also feasible, for example, when a public355venue or a concert promoter wants to have a dedicated slice for a weekend-longfestival and optimize it for streaming high-quality video and music data.While it is possible to envision a network slice to support a particular IoTservice throughout the fog hierarchy, it is the ability to combine and process vastamounts of data in the cloud, originating from multiple IoT services through360the fog, and to apply machine learning techniques to those data, that opensup potentially revolutionary progress in a myriad of fields, from healthcare toprecision agriculture.Edge devices in IoT are often resource-constrained in energy storage andprocessing capabilities.365The combination of cloud and fog computing pro-vides some relief to these limitations, m

ters are often physically and/or logically distant from the cloud client, implying 20 communication and data transfers to traverse multiple hops, which introduces delays and consumes network bandwidth of edge and core networks [2]. The widespread adoption of