Category: web-scale / cloud-native

Posted on

Progressive Transformation – A Reference Architecture

IDC defines Progressive Transformation as the modernisation of systems through gradual replacement of technology through a prioritised catalogue of business functionality. It leverages open APIs, agile development and cloud architectures.

I think IDC’s definition is a good start, but it needs to go further. It needs to extend into capabilities not just functionality. Successful transformation considers the evolution of people (incl. skills & culture) as well as the process dimension. Let me share what I’ve witnessed over the past few years across different industries.

This reference architecture brings together several concepts:

Context – Why does the world need this?

Over the last 4 years, while working across different industries in the ‘digital’ space I have witnessed the necessity to compete at speed while retaining a high level of quality. The learnings and observations are compiled into this reference architecture. It addresses the need for incumbent organisations to step up as nimble digital disrupters enter industries with the aim to compete with specific products or services of the incumbent’s value chain. Being small, nimble and fast and without slow-to-change legacy environments, slowly but surely those disrupters are reducing the customer value proposition and share of wallet of existing players.
This reference architecture helps established industry players to compete against new nimble entrants.

Intro – A Historic View

‘Digitally Infused Business Transformation’ is going on already for quite a while. Progressive means continually and ongoing, not big bang. Interestingly, 10 to 15 years ago we weren’t really talking about ‘Digital Transformation’ that much – just projects really – but whenever I see technology used to alter (and hopefully improve) customer or employee experience – meaning it changes people, process AND technology then I deem it Digital Transformation.

What has changed though the most is
a) Of course the technologies used but also
b) The expected outcomes in terms of user experience and speed of delivery of new features.
The reason this is important is because it directly alters the solution architectures, designs and impacts the way of working of delivery teams. Less and less are people willing to wait months for an important new feature.

A quick history detour

Around the year 2000 we developed web portals in ASP/PHP and a database running on clustered servers, business applications were written in J2EE using EJBs/JPA and hibernate. In 2005, I developed mobile applications using the .NET compact framework. Even though high cohesion and loose coupling has always been a design principle, back then we didn’t really talk about APIs, distributed integration, and microservices. Gantt charts and project plans were a must. The Agile Manifesto was born 2001. Domain Driven Design just came to live around 2003 and Linux Containers around 2008. In 2004, my team implemented for the first time ‘daily build and smoke tests’ which I consider the predecessor of today’s CI/CD.
All those individual evolutions in combination with recognising organisation culture as a key enabler to create high performing organisations have led to a perfect storm which today drives Digital Transformation.

Enough reminiscing.

Conceptual Reference Architecture

I am now introducing a conceptual reference architecture to enable IDC’s Progressive Transformation which my colleagues and I have applied and used as a starting point within successful customer engagements. This reference architecture has even found its way into strategy documents in our client base.

Before I start I’d like to recognise different architectural layers. From top to bottom these are:

  • Contextual Architecture
    Describes the wider context of a system/solution for example an industry, geography or regulatory requirements
  • Conceptual Architecture
    Captures specific architectural concepts that establish the guardrails for the logical and physical architecture as well as the lower level designs
  • Logical Architecture
    Details further logical components of the conceptual architecture, for example internal and external APIs
  • Physical Architecture
    Maps the logical components to the physical infrastructure components, for example the components are running on multiple Kubernetes clusters or servers in different regions.
Reference Architecture for Progressive Transformation

I leave the topic of Day 2 operational concerns, Agile DevOps teams and Service Reliability Engineering for core platforms and systems out for this post, but they are nevertheless important. That’s essentially what the yellow box on the right is for.

The 3 red bars are part of the fast cadence, fast rate of change paradigm, where Agile and DevOps enabled teams work with the customers to drive the desired features and functions through fast feedback loops. Those feature teams are responsible for the development as well as the running of those services (DevOps) teams. There is no throwing over the fence on Friday afternoon. Organisations often prefer a microservices architecture, although it’s not mandatory.

The Mode 2 architecture layers

  • Experience Layer
    APIs that define the services endpoints across all channels, e.g. for web or mobile applications. A BFF shim would be part of such a layer.
  • Business Logic Layer
    The business logic layer encapsulates organisational business logic. This can be either straight through code of Java/PHP/C++/.NET applications or through business process or business rules engines.
  • The ‘Smart’ Data Services Layer
    This one I call smart, because there needs to be quite some thinking behind how you engineer this layer. Macquarie Bank for example has put a Cassandra database into this layer. This often leads to a physical data replication and additional data synchronisation efforts. That’s why the reference architecture recommends a virtualised data layer backed by clustered in-memory data grid for speed and de-duplication of physical data. This is mostly for READ operations, while the transactions are still processed through the backends. A Kafka backed event stream can ensure the backends are not overloaded.
  • Semantic integration between those layers is handled through distributed, lightweight, containerised integrations instead of a monolithic ESB appliance. Service dependencies are handled through a Service Mesh such as Istio.

Most importantly is that those different architectural components are hosted on a platform which abstracts low level concerns such as storage, networking, compute, O/S away for the feature teams to focus on customer requirements with a starting point as close as possible to the customer aka as high as possible up the value chain.

Mode 1

Underneath we have the Mode 1 layer. This is made up of mostly existing monolithic middleware components and core systems which feed the mode 2 layer above. That said, it can also be your Salesforce CRM system that contains important data that service/product teams need to draw upon. Those systems are generally maintained by traditional ops teams. Upgrades and migrations are often executed in a traditional waterfall plan-do Both systems and ops teams are commissioned to keep the lights on, not for speed of change. Business critical information assets need to be backup-ed and restored, batch processes and maintenance routines run (processes). This Mode 1 paradigm is also important to not have all people at once change the way they work. Agile is less frequent in those teams, although in progressive organisations I see Mode 1 teams generate upward pressure to their managers as they too want to use new tools, technologies and ways of working. This is where automation and a road map towards service reliability engineering (SRE) can become important to keep growth mindset staff engaged and progressing.

To summarise, those concepts marry new modern ways of application design (microservices, Domain Driven Design) and modern ways of working (DevOps & Agile) with existing legacy systems and mode 1 operations. This combination allows incumbent industry players to compete with digital disrupters in their own or even adjacent industries.

Keen to get your thoughts,
Andreas

Posted on

Not all Open Source is made Equal

It is quite astounding to see how many proprietary software vendors have come out now claiming their products are Open Source.

If you walk back down memory lane, some of those very same vendors made fun of Open Source software not too long ago (including myself, but I won’t mention that). It’s OK though, everyone has a right to learn and grow at their own speed.

Whereas it’s quite clear why it’s great to do ‘Open Source’, consumers need to be aware of some nuances of Open Source Software (OSS), as it can impact ROI, speed of innovation and can even mean lock-in. Plus it’s where your missing innovation budget might be hiding.

Below are some aspects to consider when choosing your OSS stack and provider:

What? Why is this important?
Project VS Product Understand the difference between Open Source Software (OSS) projects versus product. A project is driven by the community, it’s the latest and greatest. An OSS provider should take the OSS project and make it enterprise ready, security harden it, ensure backwards compatibility, do testing and convert it into a supported OSS product and release bug/security fixes and put enhancements back into the OSS project.
An OSS vendor fully supports an Open Source product, whereas an Open Source project is supported by the community on a best-effort basis. Enterprises usually need a fully supported product and can’t rely on a best-effort support.
Enterprise Readiness Does the Open Source product provider understand ‘Enterprise Readiness’ such as 24×7 support, offer extended lifecycle support (ELS), patching, upgrades, updates, security response team with quick fix time SLAs, training, mentoring and Professional / Consulting Service offerings, a rich partner ecosystem and certified hardware and software stacks? If any of above are non-existent you need to question as to why and ascertain whether the OSS provider offers solutions you can rely on to run business critical functions.
Product components Are all product components open sourced or do specific components (eg connectors, add-ons, plug-ins) which you require, need a proprietary license to run? What happens if the license expires? Can you reboot and the software will still be running after the license has expired?
Upstream First Does the open source software product provider give the fixes back to the community project? By doing so the community version will be in alignment with the product. Not doing so is basically forking and the power of community powered innovation gets lost as the community will produce a different, non-compatible version.
Any vendor who does not contribute enhancements back into the upstream project is not open source. Open Source is about give and take, not giving back to the community is just take.
Stability VS Latest Features Does the Open Source Software provider guarantee backwards compatibility, migration paths or is there a risk of being forced to upgrade and re-engineer your business applications because the latest feature release is not backwards compatible? From a business perspective you want to be in charge when deciding when to upgrade and you should prefer a OSS provider who understands why stability is more important than the latest hype or feature.
Support life-cycle How long is the OSS providers’ support lifecycle – 1 year, 3 years, 10 years? You don’t want to be forced to upgrade every 1, 2 or 3 years because bug fixes are only available in the latest version.
Community viability In that instance size does matter. How large is the community behind an OSS project? Is the community perhaps just the vendor staff or is there a real non-vendor funded community driving the project innovations? How active is the community? What’s the commit frequency? How mature is the community behind a project?

If there is no real community behind an OSS project you are losing out on true community innovation power with slower innovation and bug-fix cycles than your competitor who uses a product which is based on large OSS community projects. This can affect your business innovation speed.
Project direction Is the project taking on established industry trends and standards such as packaging formats (eg Docker), patterns (eg EAI), platforms (PaaS/CaaS) or orchestration frameworks (Kubernetes)? This will determine whether your investments in the OSS product are paying off in the mid/long term future. Remember innovation happens way faster than ever before and is accelerating.
Ecosystem / stack support Does your OSS provider have other offerings up and down the software layer stack? Good News if they do, as it allows you to have a strategic partner you can work with and who understands your business, instead of calling many different vendors/service providers to get a production issue fixed, and none of them understanding the impact on your business or worse finger pointing and saying ‘not our fault’.
Subscriptions VS Licensing Do you need a license for your OSS product to run or is the subscription you buy merely to access the OSS providers auxiliary services such as updates, patching, security fixes, support etc? This is important because requiring a license to run a OSS product is similar to a vendor lock-in you get with a proprietary software vendor. A subscription in the contrary does not lock you in and you are free to move anytime. Longer license terms (eg 3 to 5 years) also significantly affect the accounting and potentially the OPEX / CAPEX classification of your software assets depending on your industry specific accounting rules.

A subscription is deemed a services, and as such is classified as OPEX. A license on the other hand is considered CAPEX. This is an important distinction, since your organisation may find itself under considerable CAPEX pressure, while the ability to reduce your fixed OPEX costs through the strategic deployment of OSS subscriptions could result in some significant cost-savings.

This is where your missing budget for innovation can come from.

I hope that helps to select the right partnerships in the Open Source space to make your business thrive in the Digital Economy.

Please let me know if you want different aspects added.

Andreas

Posted on

Mobility Platform Considerations

Mobile is Dead! Long live Mobile!

Mobile First was yesterday, well sort of. It is still true that you don’t need to execute massive enterprise transformation programs or backend system (ERP, CRM, HR, etc) modernisations before you develop your Mobility enterprise capability.

However the learnings from the early Mobility adopters is that the management (not the build and test!) of more and more mobile apps becomes exponential (not linear as expected, that is) more expensive. Different technologies, different deployment approaches, etc. In simple terms: some out of the box mobile apps that ship with your shiny HR system and the in-house developed field service mobile app which is seen as a competitive differentiator are just…well…different, and hence need some different tender love and care. That’s when a Mobility Platform strategy comes in handy.

Once you have determined that you best ride with a mobility platform (either due to the Gartner rule of 3) or due to any other means such as a business case, I have documented things to consider.

The mobility ice berg

As my colleague Wayne B. once said every topic needs an iceberg and here’s the mobility iceberg. It’s by no means complete, but it shows you that the actual mobile app is only a tiny component compared to what Enterprises need to look out for when deploying and managing mobile applications.  And that is true for both consumer facing and internal employee apps. With or without BYOD or a defined MAM/MDM approach.

Market Research Have I got a good overview of the market, e.g. through Forrester, Gartner, blogs and forums?
Peer review in other organisations How do Digital Transformation agents in other organisations address this topic?
Hybrid Infrastructure Cloud / Hosting Can the platform incl design and run-time elements be easily moved between on-prem, public infrastructure and support both deployments (hybrid) at the same time across Dev/Test/UAT/PROD?
How can the number of platform infrastructure nodes be extended or decreased to support scalability requirements? Is it dedicated or multi-tenant?
Licensing / subscriptions What is the cost structure regarding users / nodes / applications. How does on-boarding of new applications and/or users or the need for more compute / memory / storage affect pricing? Are there any user, back end services or app restrictions. What is the definition of a user (mobile end user, developer, tester etc)?
Software/solution Development Lifecycle (SDLC) How do web-scale/cloud native applications move through the SDLC? Is it code or binary based? Do I have a choice? Can I implement emergency deploy scenarios? Can I use web-hooks to trigger builds? Can I do a Source to Image build?
CI / CD Does the platform support my CI / CD processes and pipeline? How will it integrate?
API How powerful is the API? What does it support in terms of build/design and run-time (DevOps)? What DevOps, CI/CD processes does the platform support – is there a restriction? How powerful is the API?
Mobile App – Target Platforms Can you develop and manage native, hybrid and web apps? How are web apps hosted – do you need additional servers?
Frameworks What mobile application development frameworks are supported? Is there a restriction on what can/cannot be supported? How are those frameworks updated and what’s the frequency of those updates? What happens if new frameworks come out – can those be onboarded easiliy onto the platform?
 MDM – Mobile Device Management What exisiting MDM suites are supported? HOw does integration with MDM solutions work? What is pushed to those MDM solution – code, binaries?
Distribution How are mobile applications distributed? Is there a private AppStore, QRCode/URL for downloading apps for testing easily? Are Apple, Google, Microsoft, Blackberry app stores supported? How can the platform integrate with those platforms?
Governance Is it compatible with our internal project/release processes and the associated project governance and delivery model?
Collaboration & Project Level Isolation How can project teams members collaborate across the necessary dev & test roles across  UX/UI design, native Android/iOS, Hybrid framework, web, business logic and back end integration ?

How are users, teams, backend services, repositories, code and applications isolated or shared on a per project, app and backend service basis and across applications? How can people & teams collaborate local and remotely?
Tooling How can the platform support existing development, test, deployment tools and tool chains? Do I know what the existing or target build/deploy process looks like? Does the platform force any specific tools?
Source Code How is source code managed (SCM) around the platform? What existing SCM components are currently being used within my organisation?
Backend System Access and Integration How are existing Enterprise services and integrations accessed on a per application, project or user bases?
How does the platform support existent architectural concepts such as Microservices, SOA, transactions (ACID/BASE), APIs?
Bandwidth / Throttling How does the platform manage bandwidth constraints on mobile networks?
Re-usability What is the level of re-usability across applications, code, patterns, reference architectures, libraries, corporate repositories and services?
Data backend / Storage What data backends are supported out of the box? How are non-out of the box data backends supported or integrated?
Business Case What is my business case timeframe? How is my ROI calculated? What is the TCO over the x years of timeline? Do I need capitial funding or can I run a OPEX model? Is there a license + 20% maintenance or annual subscription pricing?
Support Which part of the build/run-time stack is supported by the vendor, i.e. Cloud infrastructure, Operating System, application platform, application run-time? Is the vendor support enterprise ready and 24/7?
Vendor Stability Is the vendor financially stable? Is it self (profit) or VC funded?  What’s the revenue/profit per year? What do the financial analyst rate the vendor? How long has the vendor been around? How does the vendor go about R&D and select the next new PaaS  features?
References How many existing customers are there? How many success stories are there?
Application Deployment – different version and version upgrades How does the platform support different versions of an an mobile app and services connecting depending on functionality and compatibility?
Architecture – support and constraints How does the platform support your reference architectures (eg backend service integration, APIs, business logic and front end? Is my desired deployment architecture (onPrem, multi-AZ, multi region) supported?
Data Security What data security (in transit / at rest) is supported and how?
Authentication / Authorisation What protocols are supported? Is MFA supported? Are API keys supported? Can I registere and identify specific devices?
Compliance How does the platform support the necessary compliance requirements such as SOX, ISO27001, Common Criteria?
Existing skills and change management How can the platform support re-use of existing skills sets and help minimise the organisational change management component? Does the vendor provide trainings (on-site, online, class room) and certifications?
Implementation support Does the vendor offer implementation services? Does the vendor have a strong partner eco system? How expensive and available are those resources in the market place?
Scalability How does the platform allow to scale in high-demand and off-peak scenarios? What work load density is realistic and supported?
Technology Are there any specific technologies mandatory I need to train on to use the platform?
Platform run time How many compute nodes, memory, CPU, storage does the platform need to run on based on my load scenarios?
Platform Architecture What are the architectural components of the platform? Are there any proprietary components that lock me in?

 

Posted on

Platform-as-a-Service (PaaS) Considerations

As of late, I find myself involved in interesting conversations around ‘Digital Disruption’. Eventually we will have to drop ‘disruption’ as change, constant learning and clear communication will be the new normal for businesses to survive.

We can change culture through using different tools. Enablers to instill a culture and mind set of fast change are concepts like DevOps and PaaS. Buzz word bingo aside, for me that means web-scale, cloud-native application architectures, development and deployment process readiness, multiple deliveries per day, CI/CD tooling, tool chains and executive sponsorship to ‘Make it Happen!‘ (which really means to shortcut the organisational change management bureaucracy, politics and internal stakeholder management efforts, which are ultimately roadblocks and threats to the survival of a company in the digital age). And that’s exactly what we see in our customer base who are successfully undertaking Digital Transformation.

From there on forward it becomes obvious that we want to be ‘no worries’ as much as possible about the layers underneath whatever makes up a customer consumable service/function/feature/application to be developed and operationalised (DevOps). Below an example of what I call an Enterprise Ready Container Reference Architecture.

screen-shot-2016-11-07-at-10-34-38The only reason you want PaaS is to make your life easier. Easier can mean making you faster, more scaleable, reliable and/or with higher levels of quality. Nowadays it’s not the big eating the small, it’s the fast eating the slow, for breakfast, lunch, dinner and dessert at your next all-inclusive holiday accomodation you booked through Air BnB (which made the agent you booked through last time go out of business). Therefore what you do not want is to invest in proprietary technology (increases risk of lock-in and technical debt) or having to deal with technical issues you didn’t have to think about before you went down the DIY PaaS path (for example container security, orchestration, scalability algorithms, sourcing secure container images or container networking).

A PaaS conversation has many different angles to it, hence it’s absolutely vital to see through empty Marketing promises, get a comprehensive picture and focus on what’s important to your organisation. A PaaS should be fit for purpose for your business model. Your architecture should not be driven by product features nor vendors (search also for opinionated, structured and unstructured PaaS) but by your business needs.

Because there are many offerings out there that call themselves ‘enterprise ready’ even when they are not, I compiled a list of questions to ask when choosing your PaaS.

Feedback is always welcome.

Enjoy,
Andreas

PS. The best way to move forward I believe is to define a Minimum Viable Product from front end/API to backend integration and see the entire SDLC in action around your PaaS. Marketing slides, thick strategy papers and multi-month planning cycles are not a focus area within the successful Digital Transformation programmes I have witnessed.

Market Research Have I got a good overview of the market, e.g. through Forrester, Gartner, blogs and forums?
Peer review in other organisations How do Digital Transformation agents in other organisations address this topic?
Hybrid Infrastructure Cloud / Hosting Can the run-time be easily moved between on-prem, public infrastructure and support both deployments (hybrid) at the same time across Dev/Test/UAT/PROD?
How can the number of platform infrastructure nodes be extended or decreased?
Licensing / subscriptions What is the cost structure regarding users / nodes / applications. How does on-boarding of new applications and/or users or the need for more compute / memory / storage affect pricing?
Software/solution Development Lifecycle (SDLC) How do web-scale/cloud native applications move through the SDLC? Is it code or binary based? Do I have a choice? Can I implement emergency deploy scenarios? Can I use web-hooks to trigger builds? Can I do a Source to Image build?
CI / CD Does the platform support my CI / CD processes and pipeline? How will it integrate?
API How powerful is the API? What does it support in terms of build/design and run-time (DevOps)? What DevOps, CI/CD processes does the platform support – is there a restriction? How powerful is the API?
Application – Target Platforms What target platforms does it support natively? How are web applications scaled on port 80/443?
Frameworks What application development frameworks are supported? Is there a restriction on what can/cannot be supported.
Cloud Management & Monitoring across PaaS, Containers and IaaS How do you manage your heterogenous IaaS providers (AWS, onPrem, OpenStack, Google, VMWare) and containers through a single pane of glass? Do you have/need a consolidating monitoring solution?
Distribution How are the applications exposed to the public? Across different geos?
Governance Is it compatible with our internal project/release processes and the associated project governance and delivery model?
Project Level Isolation How are users, teams, backend services, repositories, code and applications isolated or shared on a per project basis and across applications? How can people & teams collaborate?
Tooling How can the platform support existing development, test, deployment tools and tool chains? Do I know what the existing or target build/deploy process looks like?
Source Code How is source code managed (SCM) around the platform? What existing SCM components are currently being used within my organisation?
Backend System Access and Integration How are Enterprise services accessed on a per application, project or user bases?
How does the platform support existent architectural concepts such as Microservices, SOA, BASE, API Management and Enterprise Service Bus?
Bandwidth / Throttling How does the platform manage bandwidth constraints?
Re-usability What is the level of re-usability across applications, code, patterns, reference architectures, runtime images, libraries, corporate repositories and services?
Data backend / Storage What data backends are supported out of the box? How are non-out of the box data backends supported or integrated? What storage options do I have? Is my choice of storage supported? Is storage replication supported? Is storage assigned per application, project, container or platform wide?
Business Case What is my business case timeframe? How is my ROI calculated? What is the TCO across entire lifetime? Do I need to capitalise the cost or can I run a OPEX model – do I have a choice?
Support Which part of the build/run-time stack is supported by the vendor, i.e. Cloud infrastructure certification, Operating System, container run time, orchestration engine, application platform, application run-time? Is the vendor support enterprise ready and 24/7? What are the vendors response/fix time SLAs?
Vendor Stability Is the vendor financially stable? Is it self (profit) or VC funded?  What’s the revenue/profit per year? How do the financial analysts rate the vendor? How long has the vendor been around? How does the vendor go about R&D and select the next new PaaS  features? Is the vendor Enterprise and/or consumer/developer focussed?
References How many existing customers are there? How many success stories and references are there?
Application Deployment – different version and version upgrades How does the platform support different versions of an application connecting to different end points depending on functionality? Are blue/green deployments supported? How can I roll back a failed deployment?

What programming languages are supported and how can you add additional languages to the platform?
Architecture – support and constraints How does the platform support your reference architectures (eg backend service integration, APIs, business logic and front end, BASE, SOA) while observing loose coupling and high cohesion? Is my desired deployment architecture (onPrem, multi-AZ, multi region) supported? How is automatic data replication supported across multiple nodes in different geographies? What storage options do I have?
Security What data security (in transit / at rest) is supported and how?
What run-time stack security is available from Operating System, platform, to container? Who is patching security issues?
Authentication / Authorisation What protocols are supported? Is MFA supported?
Compliance How does the platform support the necessary compliance requirements such as SOX, ISO27001, Common Criteria?
Existing skills and change management How can the platform support re-use of existing skills sets and help minimise the organisational change management component? Does the vendor provide trainings (on-site, online, class room) and certifications? Will I create a proprietary skill set or is there an Open Source community available to me?
Implementation support Does the vendor offer implementation services? Does the vendor have a strong partner eco system? How expensive and available are those resources in the market place?
Quality of Service – Scalability & Clustering How does the platform allow to scale in high-demand and off-peak scenarios? What work load density is realistic and supported?

What elements are looking after QoS concerns and how mature and supported are those?
Technology Are there any specific, non-standard technologies mandatory I need to train on to use the platform?
Platform run time How many compute nodes, memory, CPU, storage does the platform need to run on? What is the level of workload consolidation?
Platform Architecture What are the architectural components of the platform? Are there any proprietary components that lock you in?
Networking Is the networking architecture flexible, e.g. is Software-Defined-Networking utilised, if so is the implementation proprietary or supported by a large and active community?
Orchestration How is container/microservices orchestration implemented? Proprietary or standards based?
Applications Are there vendor supported and certified container image registries available? What does the update/notification mechanism look like if new images with bug/security fixed become available? Which components of my application stack is supported (Business rules, data grid, Data Virtualisation, application server, API management, Mobility) and maintained (security fixes, upgrades, patches) by the vendor?