How DevOps Could Have Saved Equifax

+
-
0
How DevOps Could Have Saved Equifax

How DevOps Could Have Saved Equifax
by Bob Aiello

Equifax is the latest large firm to make unwanted headlines due to exposure of clients’ personal data; a reported 143 million people may have had their Social Security numbers, birth dates, credit card numbers and other personal information stolen. According to published accounts, the breach occurred through a vulnerability in the Apache Struts web framework which is used by many organizations. The incident was an embarrassment to a company whose entire business revolves around providing a clear, and presumably confidential, financial profile of consumers that lenders and other businesses use to make credit decisions.

Large organizations often have hundreds of major systems using thousands of commercial and open source components – each of which could potentially have a security vulnerability. The Apache organization issued a statement about the most recent incident. There were also many alerts issued about the potential risks in the Apache Struts framework, but large organizations which receive alerts via Common Vulnerabilities and Exposures (CVEs) and VulnDB may find it very difficult to identify exactly which software components are vulnerable to attack and be unable to quickly fix the problem and/or deploy the updated code that prevents hackers from exploiting known vulnerabilities.

So how best to handle these scenarios in large organizations?

The first step is to get all of your code stored and baselined in a secure version control system (VCS). Then you need to be able to scan the code using any of the products on the market which can identify vulnerabilities as reported in CVEs and the VulDB database. There are costs involved with implementing an automated solution, but the cost of not doing so could be far greater.

One approach could be to clone each and every repo in your version control system (e.g. bitbucket) and then programmatically scan the baselined source code identifying the projects which contain these vulnerabilities. You can get better results if you scan code that has been compiled, as the build process may pull in additional components. But even just scanning source code will help you get the conversation involving your security experts, operations engineers and the developers who wrote the code started. Suddenly, you can find that needle lost in your haystack pretty quickly and begin taking steps to update the software. Obviously, another key ingredient is having the capability to immediately roll out that fix through a fully-automated application build, package and deployment process, what many folks are referring to as continuous delivery.

Implementing these tools and processes does take some time and effort, but, as the Equifax data breach has painfully demonstrated, effective DevOps is clearly worth it.

What is your strategy for identifying security issues buried deep in a few hundred thousand lines of code? It is actually not that hard to fix this issue as long as you can work across development, operations and other stakeholders to implement effective CM best practices including:

Source Code Management
Build Engineering
Environment Management
Lean and Effective Change Control
Release & Deployment engineering

yeah – I am saying that you need DevOps today!

Bob Aiello
bob.aiello@ieee.org

 

How DevOps can eliminate the risk of ransomware and cyber attacks

+
-
0
How DevOps can eliminate the risk of ransomware and cyber attacks

How DevOps can eliminate the risk of ransomware and cyber attacks
By Bob Aiello

Reports of global cyberattacks, said to have impacted more than 200,000 users in over seventy countries, have certainly garnered much attention lately. You might think that the global IT “sky” was falling and there is no possible way to protect yourself. That just isn’t true. The first thing that you need to understand is that any computer system can be hacked, including yours. In fact, you should assume that your systems will be hacked and you need to plan for how you will respond. Experts are certainly telling us all to refrain from clicking on suspicious attachments and to keep our Windows patches up-to-date. None of that advice is necessarily wrong, but it fails to address the real problem here. In order to properly avoid such devastating problems, you really need to understand the root cause. There certainly is plenty of blame to go around, starting with who made the malware tools used in the attack. There is widespread speculation that the tool used in the attack was stolen from the National Security Agency (NSA), which leads one to question whether those agencies in the business of securing our national infrastructure are really up to the job. This global cyberattack was felt by thousands of people around the world.

Hospitals across the UK were impacted, which, in turn, impacted medical care, even delaying surgical procedures. Other organizations hit were FedEx in the United States, the Spanish telecom company Telefónica, the French automaker Renault, and Deutsche Bahn, Germany’s federal railway system. I have supported many large organizations relying upon Windows servers, often running complex critical systems. Building and upgrading windows software can be very complex and that is the key consideration here. It is often not trivial to be able to rebuild a Windows machine and get to a place where you have software fully functioning as required. I have seen teams tinker with their windows build machines and actually get to a place where they simply could not build another windows machine with the same configuration. Part of the problem is that very few technology professionals really have an understanding of how Microsoft actually works, which really is the problem here. In DevOps, we need to be able to provision a server and get to a known state without having to resort to heroic efforts.

With infrastructure as code, we build and provision machines from scratch using a programmatic interface. Many use cloud-based resources which can be provisioned in minutes and taken down just as easily when no longer needed. Container-based deployments are actually taking this capability to the new level making infrastructure as code routine. From there, we can then establish the deployment pipeline and within a reasonable period of time, which enables us to deploy known baselines of code very quickly. Backing up our data is also essential and in practice, you may lose some transactions. If you are supporting a global trading system, then obviously there must be strategies to create transactions logs which can “replay” your buys and sells, restoring you to a known state.

What we need now is for corporate executives to understand the importance of having competent IT executives implement best practices capable of addressing these inevitable risks. We know how to do this. In the coming months, a group of dedicated IT professionals will be completing the first draft of an industry standard for DevOps, designed to cover many of these considerations. Let’s hope the folks running institutions from hospitals to retail chains take notice and actually commit to adopting DevOps best practices.

Parasoft – API Testing and Service Virtualization at Microsoft Build

+
-
0

Parasoft showcases new release of API Testing and Service Virtualization at Microsoft Build

Monrovia, CA & Seattle, WA – May 10, 2017 – Parasoft, the leader in software testing solutions, today announced its latest enhancements of their API testing and service virtualization solutions for the Microsoft Environment at Microsoft Build 2017, taking place May 10-12 in Seattle. Parasoft will be featuring its new functionality at booth #209. To get started and learn more, visit: http://software.parasoft.com/virtualize/microsoft/

Parasoft SOAtest and Virtualize are widely recognized as industry standard tools for enabling teams to quickly solve today’s most challenging issues, including security, performance, and test environment obstacles. In a continued effort to improve functionality and ease-of-use for customers, Parasoft has introduced new functionality and streamlined workflows to address everyday challenges that software developers and testers face.

Parasoft has focused on three key areas with this new release:

  • Broadening access to testing through the thin client interface: Greater access enables teams to quickly initiate testing projects, facilitate correlation and collaboration, and seamlessly tie test scenarios to environments.
  • Solving data challenges through enhanced workflows: Providing quick and simple access to test data helps test designers create move efficient and effective tests.
  • Shift-left performance testing: Early-stage performance testing is available by reusing existing test artifacts in performance tests and reviewing results in the Web-enabled dashboard.

To learn more about Parasoft’s offering, please visit:

About Parasoft

Parasoft provides innovative tools that automate time-consuming testing tasks and provide management with intelligent analytics necessary to focus on what matters. Parasoft’s technologies reduce the time, effort, and cost of delivering secure, reliable, and compliant software, by integrating static and runtime analysis; unit, functional, and API testing; and service virtualization. Parasoft supports software organizations as they develop and deploy applications in the embedded, enterprise, and IoT markets. With developer testing tools, manager reporting/analytics, and executive dashboarding, Parasoft enables organizations to succeed in today’s most strategic development initiatives — agile, continuous testing, DevOps, and security.

Behaviorally Speaking: Creating the Deployment Pipeline

+
-
0
Behaviorally Speaking: Creating the Deployment Pipeline

Creating the deployment pipeline.
By Bob Aiello

Continuous Delivery (CD) depends upon the successful implementation of a fully automated deployment pipeline. As a practitioner, I have heard many folks refer to CD in overly simplistic terms, creating a naive image of deployments being fully automated as a simple push button. The truth is that continuous delivery is anything but simple and it takes some effort to create a fully automated deployment pipeline. Continuous Delivery touches every aspect of the application lifecycle and is best created in an iterative manner. This article will help you get started with creating your fully automated continuous delivery pipeline.

The first thing that you need to understand about Continuous Delivery is that it is – well – continuous. This means that you need to consider every step of the process to create your application build, package and deployment. Some folks refer to continuous deployment and continuous delivery in the same sentence. In fact, I have seen DevOps thought leaders use these terms interchangeably. Continuous Deployment refers to being able to push changes as often as necessary. Continuous Deployment can be disruptive in that users are often not ready for a change. You should use continuous deployment when you must push out an urgent security patch – with or without the user’s consent. Continuous Delivery is more subtle in that changes are technically pushed to the target environment, but are hidden through a technique known as feature toggle where changes are hidden until they are ready to be exposed to end users. Continuous delivery helps reduce risk as changes are technically deployed and can be tested to some degree. Continuous delivery is less disruptive than continuous deployment because continuous deployment pushes changes whether or not the user is ready to accept them.

The deployment pipeline is the automated procedure that implements continuous delivery and continuous deployment. Most often, deployment procedures consist of shell scripts, which ensure that each step of the build, package and deployment is successfully completed without any errors. If you were to look at my scripts, you would see that more than half the code is testing each step and then logging the results. Mistakes happen and it is very common for one step in the deployment automation to not complete as expected. The deployment pipeline completely eliminates manual steps, but recognizes the reality that sometimes problems occur. If a step does not complete as expected, you want the deployment to fail immediately and notify the operator as to what has occurred. One challenge that often arises is how to handle reviewing and approving proposed changes. Agile and lean change control is an absolute must-have.

The secret to effective change control is to identify which changes are low risk and can be categorized as being preapproved. The remaining changes, by definition are a little more complicated, and should be reviewed and assessed for technical risk. The DevOps approach is to ensure that all of the subject matter experts (SMEs) are involved with the technical review process. It has been my experience that too often change control resembles a phone-tag game where change managers “represent” changes which they barely understand instead of ensuring that the SMEs are fully engaged with the technical review process. The deployment pipeline relies upon effective change control which employs agile and lean principles.

The deployment pipeline should pull source code from the version control system (VCS) based upon labeled (or tagged) baselines and built via a fully automated procedure – most often using Ant, Maven, Make or other build scripting language. Continuous integration servers such as Jenkins initiate the build, often triggered by the code being committed into a version control system. I try to script every single step using Ruby, Python or the available command line tools such as bash or, on windows, PowerShell. Remember that we build once and then deploy the same built components to each environment using the same automated procedures. The build process should also automatically generate the SHA1 or MD5 hashes which can be used to verify that each configuration item was successfully deployed to its target location and also later used to identify any unauthorized changes.

Infrastructure should always be built using automated procedures which is known as infrastructure as code. Managing configurations and environment dependencies also should be performed via automated procedures. It is important that deployment procedures be reliable and verifiable while also logging each step as it completes. The procedures should yield the same results no matter how many times they are run which is known as being idempotent.

Creating the deployment pipeline is not trivial and you may need to complete this effort in steps. I always focus on starting with attended automation where I do my best to automate each step and require that an operator look at the screen and verify that each step has completed successfully and then press enter. Over time, I may know enough to be able to fully automate the build, package and deployment as that magical “push button” that we all strive for, but in the meantime the steps I have outlined will significantly improve the reliability of your deployment process – eliminating many potential sources of error.

If you want to succeed in creating the deployment pipeline then start in small steps. Eliminate any possible sources of error. Manual steps will result in human error. Taking an agile iterative approach to creating the fully automated pipeline will help you deliver changes as often as necessary while ensuring security and reliability. May sure that you drop me a line and share your best practices for creating the deployment pipeline!

Imitation is Limitation – Why Your Agile and DevOps Transformations are Failing

+
-
0

Imitation is Limitation – Why Your Agile and DevOps Transformations are Failing

By Nicole Bryan

If you’re a business or IT leader trying to compete in a digital world, you need to leverage your software delivery capabilities to the max to stay competitive. If you don’t, your organization is ripe for digital disruption from younger, more digital-centric companies.

To address this threat, you’ve probably adopted widely publicized Agile and DevOps practices to enhance your software delivery. Perhaps you or one of your team were inspired by a dazzling presentation at a tech conference that implied, “If you copy this model, you can enjoy the same success!”

But tread softly. There is no ‘one-size-fits-all’ approach to Agile and DevOps, and imitating the successful transformations of Facebook, Netflix, Airbnb, et al. will not necessarily improve your capability to quickly deliver quality software. In fact, it may be detrimental, leading to more bottlenecks and waste that could cost your business millions in lost productivity.

Why is there no silver bullet or magical blueprint? Largely it’s because you’re not the same as those digital disruptors. In fact, you’re distinctly different businesses, each with its own unique software ecosystems. These nuances must be understood if you’re to have any success with your digital transformation.

To understand why, it’s important to remind ourselves of the core values of Agile and DevOps. Both are about delivering value to the customer and for the business, and regard operational waste as the scourge of delivering consistent end-to-end value. With this in mind, let’s look at how waste is created, and value lost, when a large organization tries to scale its transformations.

  • Application-critical

Digital-native companies can adopt a ‘trial and error’ approach to building new software innovations, relaxed in the knowledge that it’s not the end of the world if there’s a bug in the system (as it can be easily rectified). Whereas larger organizations, such as banks or healthcare providers, require a more diligent approach. They will be using heavyweight tools to ensure rigorous scanning of requirements, builds and tests to limit the risk of software downtime so they can always access bank accounts and find medical records. Being tied to such influential legacy tools within the workflow can slow down the speed with at which teams can operate – unless these tools are working in harmony with all the other tools in the lifecycle, which they’re not naturally designed to do.

  • Audit

Back in the day, companies such as Facebook and LinkedIn were private and didn’t need to worry about regulatory or corporate policies. Now that they are public, they must adhere to strict rules and regulations, as do most established organizations. Audits are a huge part of industries such as financial and healthcare, and the software delivery environment must document every activity within the software lifecycle. More teams and tools mean more elements that need to be consistently recorded across all systems and databases so there’s ‘one source of truth.’ Without an automated flow of information into a centralized point, this process is a time-consuming manual entry task for big enterprises, generating huge amounts of waste on non-value work. Smaller private companies don’t have such pressing audit concerns, if any at all, meaning more time spent on value-added activities.

  • Developer pool

Large global organizations tend to have thousands more developers than their younger competitors. All of these developers have their preferred tool and Agile methodology, which creates much conflict and discourse. Meanwhile, digital-native companies tend to work from the same place – the same tool, same methodology and a clear, shared goal. A connected software value stream can help organizations create unity and increase understanding in every aspect of the software development and delivery lifecycle, removing waste and keeping the focus on value-added work.

  • Partner concerns

To operate at a high level across the board, parts of the business are outsourced to third parties to develop components or even whole non-trivial applications. Unfortunately, these third parties are unlikely to share the same toolset as their client, creating a discontinuity of information that causes unnecessary friction. Again, digital-centric companies are unlikely to have such concerns, as they won’t be delegating or outsourcing on such a large scale. With a system in place to connect an organization’s toolchain with its partners, the flow of information can be instant and controlled.

These are just a few of the key encumbrances that larger organizations must contest with, and that heavily influence any Agile and DevOps transformation. By this point, you may be thinking “Well, what’s the point? The game’s over. Throw in the towel!” But don’t give up, because there is a way forward.

Many transformations fail because the flow of project-critical information between key stakeholders is too slow, damaged or AWOL. This means much waste, a plethora of doomed projects and lot of unhappy, disengaged employees.

This information must be flowed across tools, teams, disciplines, organizations and partners, as the data’s very creation was intended to do. To do this, you need to integrate your software lifecycle, which resolves any conflict or friction caused by scaling tool deployments and projects.

Not only will this remove the barriers between tools and teams, helping them to work together and enhancing their individual value, but it will enable the enterprise to easily expand and manage its tool landscape. And it will result in a software ecosystem that is tailored to an individual organization’s needs and that supports customer requirements.

By creating an integrated software value stream, you create a robust backbone with which to scale Agile and DevOps transformations, enabling your organization to compete (and innovate) in a digital world.

Nicole Bryan is Vice President, Product Management at Tasktop Technologies. Nicole has extensive experience in software and product development, focused primarily on bringing data visualization and human considerations to the forefront of Application Lifecycle Management. Most recently, she served as director of product management at Borland Software/Micro Focus, where she was responsible for creating a new Agile development management tool. Prior to Borland, she was a director at the New York Stock Exchange (NYSE) Regulatory Division, where she managed some of the first Agile project teams at the NYSE, and VP of engineering at OneHarbor (purchased by National City Investments). Nicole holds a Master of Science in Computer Science from DePaul University. She is passionate about improving how software is created and delivered – making the experience enjoyable, fun and yes, even delightful.

 

Test Environment Management: The Secret Ingredient for DevOps

+
-
0

Test Environment Management: The Secret Ingredient for DevOps

 

Nikhil Kaul, Senior Product Marketing Manager at SmartBear Software

The rapid adoption of DevOps necessitates continuous delivery and a shift toward higher levels of test automation. As a result, QA teams often invest a lot of time and effort in ensuring their tests are automated as a part of the continuous testing cycle. The right test automation tools allow teams to increase test speed and coverage. In fact, a common framework agile teams use while navigating this process is the test automation pyramid – an approach that focuses on automating tests at three levels: unit, service and user interface (UI).

At the base of this pyramid is unit testing. By running more unit tests at a lower level, teams can get feedback faster and develop a solid testing base to build upon. The intermediate layer consists of API tests, which work under the UI of an application. Finally, the top of pyramid represents end-to-end UI tests.

To implement a test pyramid approach, teams spend a lot of time and energy hunting for the right test automation tools. As an example, for unit level testing, teams use JUnit or NUnit. Similarly, for API level testing, teams love SoapUI, while for UI testing, TestComplete or open source tools like Selenium could be used.

A DevOps chain can only be as strong as its weakest link. And often one area that is neglected during such a test pyramid approach is the way teams go about managing their test environments.

Test automation efforts often falter as the efficiency gained by investing in test automation is lost when teams still maintain the test infrastructure manually. In the case of UI testing, teams can spend hours provisioning, maintaining, monitoring and tearing down the underlying test environment infrastructure.

As an example, without a proper strategy, teams spend a significant portion of their testing cycles ensuring the right testing environments are available. And often, even when these environments are available, there are large discrepancies between test environments and production environments.

Traditional Ways of Managing Test Environments

 

There are three common tactics teams use to manage test environments. One of the easiest ways is to use local machines (i.e., browsers hosted locally). The second way is using virtual machines such as Amazon EC2. Finally, the last way teams go about test environments is by leveraging device labs, which primarily consists of rented or bought mobile devices.

Regardless of which of these three routes a team takes, there is manual work required to maintain and upgrade environments when new browsers, operating systems, devices or resolutions are introduced.

In addition to the manual efforts required to maintain these UI test environments, there are other challenges such as reduced speed to delivery, redundancy and discrepancies.

Speed and Agility Concerns

 

Setting up environments on local machines as a part of your testing cycle automatically slows down the entire process. QA teams frequently find running UI tests on virtual machines can help overcome up front hardware costs, but there is still a manual component involved in ensuring the right configurations are available when tests are being kicked off. Provisioning clean test environments often means spinning up new VMs with new configurations which can be time consuming and adds to the labor cost.

Parallel Execution Needs a Lot of Work

 

Having an ability to run tests in parallel across various browsers and OS combinations can be useful to speed up the testing process and increase coverage. However, often setting up parallel execution capabilities for UI tests with traditional test environment management efforts like local machines, VMs and device labs can be a painstaking process.

To run UI test on different devices, teams need to set up a hub with multiple node devices. Registering the node to the hub involves specifying arguments like browser type, version, hostname, port etc. If these arguments change or get updated, QA might have to go back to the command line or configuration file to update. Additionally, adding an iPad or an Android device as a node requires even more additional steps.

Redundant Activities

 

Setting up environments locally results in duplicated efforts as each individual has to perform the same exact activity. In case, teams are maintaining an in-house device lab, the redundant efforts associated with upgrading various browsers and OS combinations can grow exponentially. Keeping systems and devices up-to-date becomes more challenging as teams grow and new environments (web browsers, operating system version and resolutions) are added to the mix to improve coverage.

Increased Cost

 

Setting up local environments, using VMs or device labs can often result in cost overruns. In fact, costs encountered while using these options can be primarily broken down into three different buckets: device costs, labor costs and licensing costs.

Device cost represents the cost associated with having devices with the right configurations of operating systems, resolutions and browser versions.

Labor cost consists of the time and effort your operations, test or DBA team spends on maintaining, upgrading or even tearing down test environments, database servers and labs, among others.

Licensing costs include the inevitable part of using licenses of additional software to ensure environments are working as expected, both with device labs and VMs. An example, you’ll likely want to know that if and when your test environments go down. Setting up this type of notification and monitoring system for your environments requires additional spend on a third-party vendor.

UI Test Environment Management: What to Look for?

 

While looking for the best way to optimize and manage your test environments, there are four key aspects to take into consideration.

  1. Test Environments Should be Easy to Install

Easy installation helps ensure that application development and delivery (AD&D) teams don’t spend considerable amount of time with set up and configuration.

  1. Test Environments Should be Easy to Enhance

If a new browser version is released, the needs of your end-user will change accordingly – forcing you to modify your test environments. Therefore, easy to enhance test environments can thereby be really helpful, especially in a DevOps environment.

  1. Test Environments Should be Easy to Share

The easier it is to share test environments among team members, the more scalable the process becomes. The time it takes for your set of tests to complete a run can be drastically reduced by running UI tests in parallel. To do that, environments should be easy to share among team members.

  1. Test Environments Should be Easy to Tear Down

As an end user yourself, you should be able to spin down, reconfigure or reinstall your environments as needed.

The Right Way to Set Up UI Test Environments

 

Having cloud-based test environments available on demand can help overcome scalability, cost and maintenance challenges posed by use of local machines, virtual machines or an in-house testing lab.

Cloud-based environments, like TestComplete Environment Manager, give on-demand access to over 1,500 combinations of browser versions, operating systems and resolution settings. Application development and delivery teams can easily and quickly execute and report on automated UI tests across test environments without setup or configuration.

Other benefits of using cloud-based test environments include:

Easy to Install, Enhance, Share and Teardown

Cloud-based test environments are incredibly easy to manage and are accessible anytime, eliminating the need for an installation process altogether. You simply need to link those environments to your tests. Cloud-based environments are also simple to enhance. When new browser versions are released, a good cloud-based environment solution will update these, thereby eliminating your need to spend time on upgrades.

As you’re accessing a cloud-based environment through a URL, it is easy to share with anyone, enabling you to scale as needed. Tearing down environments in this case is as simple as deciding to not use the environments.

Parallel Test Execution Without Setup

The challenges associated with setting up environments for parallel execution can be easily overcome with cloud-based environments as they are built with frameworks to handle concurrency.

Report Across Environments with Zero Configurations

This is a gold mine. The reporting is built-in, so QA managers can pull metrics and see how tests are trending across operating systems, browsers or resolution settings – all in one place.

Combine Manual and Visual Tests with Automated Tests

Cloud-based test environments offer complementary benefits to automated testing which you won’t get through virtual machines, on premise systems or local devices.

For example, you will often realize that manual or visual testing is needed to uncover issues that are hard to find through automated test scripts. These include:

  1. Establishing aspects of the UI are of the right color, shape or size
  2. Uncovering overlapping elements
  3. Ensuring the appearance or usability of a website looks as expected

The right cloud-based test environment management tool will have out-of-the-box capabilities to tie these into your test automation framework.

 

Nikhil Kaul is the Senior Product Marketing Manager of Testing Products at SmartBear Software. Prior to SmartBear, Nikhil served Digité, a leading provider of Application Lifecycle Management solutions, in various roles including Senior Software Executive. There, he gained insights into software development and testing market. Nikhil received a master’s degree in business administration from Georgetown University. He is very involved in discussions taking place in development and testing communities. Follow him on Twitter @kaulnikhil or read his blog at: http://blog.smartbear.com/author/nikhil-kaul

 

 

Demystifying DevOps – Which DevOps do you mean?

+
-
0
Demystifying DevOps – Which DevOps do you mean?

Demystifying DevOps – Which DevOps do you mean?
by Uday Kumar

DevOps has been the absolute buzzword of the IT world over the last couple of years and yet its precise meaning still remains unclear. Along with Agile, this word represents a promising new wave within software development. Currently, a team of industry experts is collaborating to develop a universal standard for this ambiguous and confusing term. Until this IEEE working group presents their final document, though, those in the trenches will continue to interpret the concept with flexibility, based on a balance of experience and convenience.

Personally, viewing DevOps primarily as a new organizational culture is definitely not my cup of tea. I see greater value in the DevOps focus on maximizing team members’ creative utilization of process and tools.

I have spent a lot of time and researched many sources to determine exactly what DevOps means. Basically, there are 4 categories that are broadly associated with this term. In this blog, we will cover each of these categories.

Apart from DevOps, there are several other related buzzwords like ChatOps, CloudOps, SecOps that are now trending, based on the increasingly popular idea that software development is, by necessity, an integrative process. Apart from DevOps, the rest of these terms are out of scope for this blog.

Category 1: DevOps – Software Developers (Dev) and IT Operations (Ops) – CI & CD

DevOps is primarily defined as Collaboration, Communication and Integration between Software Developers and IT Operations, groups whose fundamental interest areas are usually different and contrasting.

Software Developers (Dev) want to focus on creating new code and applications while IT Operations (Ops) wants to focus on sustainability or quality. When the application is not working as expected, Dev often thinks “it is not my code, it’s your machine”, whereas Ops thinks “it’s not my machine, it must be your code”. DevOps is a term coined to reflect the bridge which must be built between these two teams so that new code gets deployed on the production systems smoothly without blaming one another.

In order to achieve this seamless continuity, we need to have essential process and platform in place; two essential ingredients are CI (continuous integration) and CD (continuous deployment). Tools + Automation is key. Companies will often implement Agile principles to achieve the highest degree of success with CI and CD. IBM BlueMix and Cloudbees are SAAS- based products with ready made CI and CD platforms for many different applications (especially web and mobile). Take a look at this Sonatype slideshare presentation with Architecture types to get a good overview on this topic. It is a very good reference for a deep dive into this helpful technology.

Although the majority of the people with whom I have interacted adopt this view of DevOps, many companies have started realizing that CI and CD implementation is only able to provide a partial fix to a complex problem.

Category 2 and 3: DevOps – Infrastructure (Ops) as Code (Dev)

The IT Operations team is generally responsible for managing the infrastructure of updating the software (OS /Application ) manually. Updating this configuration as part of change management is one essential step of ITIL process. It is crucial that any and all changes be documented, as well as auditable.

Virtualization and Cloud technologies have enabled team members to write the code necessary to create and manage the infrastructure, as well as to control the changes using the updated code. Docker, Puppet and Chef are taking this to the next level. Effectively, all operations that are being performed by the IT team are now getting automated. Coding is abstracting the complexity of managing the infrastructure and developers don’t have to specify required infrastructure as a guideline. The marketing materials of Puppet and Chef reflect the awareness that effective automation of Configuration Management currently falls under the DevOps umbrella. Clearly, the DevOps term provides an apt metaphor for the way to achieve the most stable environments.

Category 2: Internal Infrastructure Management for conducting different levels of testing either on Internal servers or in the Cloud (external).

This setup is quite complex especially for Product companies (not SAAS) as they need to support various product versions/variants. Along with Virtualization/Cloud, technologies like Dockers, Puppet and/or Chef will fit well. Nowadays, professionals are seeing many profiles in this area with DevOps.

Category 3: Managing the production servers especially for SAAS product (application) like Amazon, Google Apps, SalesForce.

DevOps demands continuous automation which means that scripts, instead of people, are initiating automated jobs including continuously deploying software updates. They also use automation to check the health of the system and its environment by monitoring applications, securing the applications, load balancing and dynamically provisioning new servers as needed and even ensuring automatic recovery should a problem arise. Implementation of ITIL-based solutions is also considered to fall within DevOps.

Personally, I don’t like to label automation of IT operations as being “DevOps”. The first reason is because the automation of IT operations has been around long before anyone started using the term DevOps. Addteq, has been automating process since we first opened our company over 11 years ago, long before the DevOps term caught on. Secondly, DevOps is really about improving communication, collaboration and integration between groups including business end user, development, operations among other key stakeholders.  IT automation is essential, but DevOps has an even broader focus.

Category 4: BizDevOps – Product Management (Biz), Engineering (Dev) and IT Operations (Ops)

This new term, though not yet a very popular buzzword compared to the others discussed, may just be my favorite. Probably only heard in the past 2+ years, and quickly picking up speed, this three-pronged word covers the entire value stream, starting from initial customer request straight through to delivery. Unified cross-functional teams work together (collaboration, communication and integration) in order to generate value to the customer, while simultaneously and judiciously balancing the available resources. According to some, this can be considered as Agile (Scrum/Kanban/Enterprise Agile frameworks) along with CI and CD mentioned above. IBM, CA, and Xebialabs continue to promote this categorization under the DevOps label.

 

I use the analogy of physical manufacturing to understand the software lifecycle; the development process mirrors the steps implemented in a manufacturing factory while the cross-functional value chain ( from Product Management to IT Operations ) may be thought of as a manufacturing assembly line. To achieve Enterprise Agility, the assembly line must be continuous. This assembly line is the Continuous Delivery (Development + Integration + Deployment + Testing + Service) platform which is critical to enable continuity. Integrated Release Management or Application Lifecycle Management(ALM) along with Automation are really other terms which refer to the continuous delivery platform.

At Addteq, we provide services for all the above 4 categories. Plus, we are always adding and updating based on the latest industry developments. For more details, you can refer to our extensive slide deck.

We invite customer and community input regarding this timely debate: which category do you regard as “the real DevOps”, is your understanding of DevOps included in our category list?

As there is so much confusion around this word, the next time you say DevOps to someone, it might be helpful to specify to which aspect you are referring. And if someone is discussing DevOps, then don’t hesitate to request clarification, especially during consulting and recruitment. DevOps can’t work unless everyone is on the same page.

We encourage you to add your comments or perspectives.

About the author:

Uday Kumar is Product Manager and DevOps ALM Management Consultant for Addteq, an Atlassian Platinum Solution Partner providing business solutions to enterprise clients. Uday is an Intrapreneur, providing innovation within his organization including his software development centers working towards a vision which he refers to as “Scientific Management of Software Industry”. Uday focuses on reducing all types of operational waste while achieving Operational Excellence.

Predictions for the Coming Year

+
-
0

Predictions for the Coming Year
by Bob Aiello

This is an exciting time to be part of the technology industry. The demand for complex systems is only surpassed by user expectations that new features, as well as bugfixes, can be delivered rapidly – even during normal business hours. The internet of things (IoT) has expanded the definition of connectivity to everything from your car to your washing machine. Mobile devices can help monitor your smart house, optimize your electricity and heating bills while ensuring mobile connectivity throughout your international travel. Cloud-based resources deliver remarkable scalability at low cost, enabling the analysis of big data which continuously astounds with new applications of its remarkable business intelligence capabilities. Savvy architects ensure not only an API-first, but often an API-only architecture. While there are limitless possibilities, there are also are limitless risks.

Cybersecurity has emerged as a core capability to address the challenges of cybercrime (and, for state entities, the related challenges of cyberwarfare and even cyberterrorism). The remarkable number of high-profile security incidents have damaged reputations for many firms and led many consumers to view cyber-responsibility as a must-have. In the coming year, consumers will expect companies to up their game with regard to cyber capabilities ensuring systems security and reliability. Technology professionals will need to embrace a comprehensive approach to continuous security that reaches from designing systems to be secure, through secure code analysis, to thorough penetration testing in order to confirm that applications have been not just designed, but also built, packaged and deployed to be secure.

DevOps is growing beyond deployment engineering to encompass a broader set of competencies which enable rapid delivery of business capabilities. DevOps itself will mature from the domain of internet startups to proficiencies which can align with the demands of even the most highly regulated financial services firms. DevOps is no longer just about development and operations. DevOps enables better communication and collaboration between organizational units throughout the enterprise. This journey to maturity must include an industry standard to guide the use of DevOps in firms which must adhere to audit and regulatory requirements and seasoned IT professionals are working on that effort right now. Feel free to contact me for more information on the IEEE P2675 DevOps Working Group.

Agile development has established itself as the preferred approach to software engineering. But many organizations struggle to achieve agility due to inevitable constraints – from idiosyncratic corporate culture to the complex demands of technology development efforts. As the old adage goes “nine women cannot make a baby in one month” and the realities of systems constraints often make iterative development challenging, if not near impossible. In 2017, firms will likely need to focus more on software process improvement than trying to meet the demands inherent in agile development. That’s right; I am saying to focus more on process improvement and less on proving that you are agile! Even projects which must use waterfall methodology can benefit greatly from the wisdom of the Agile Manifesto and agile principles. Our new book on Agile ALM & DevOps discusses this approach in detail. As it always has, process maturity will need to include traceability, while still achieving fully agile application lifecycle management (ALM). Organizations in highly regulated industries, including financial services, will need to implement agile ALM while still maintaining fully traceable processes.

DevOps and Agile ALM will accelerate your process developer velocity; however, your performance will be significantly impacted unless you have implemented continuous testing, including API testing and the use of service virtualization. Unit and functional testing are essential, but not sufficient, to keep up with the demands of Agile and DevOps. Agile depends upon DevOps and DevOps relies upon Continuous Testing and Continuous Security.

Tools are essential for successful software development and solid tools integration is also crucial. Vendors will continue to evolve in terms of building tools that can be integrated across the ecosystem. Vendors themselves will continue to play a key leadership role in DevOps innovation with every vendor stretching to ensure that their solutions help enable DevOps.

Bob Aiello (bob.aiello@ieee.org)

 

Enterprise DevOps and Microservices in 2017

+
-
0
Enterprise DevOps and Microservices in 2017

Enterprise DevOps and Microservices in 2017
By Anders Wallgren, CTO, Electric Cloud                                       Anders B&W

As a new year begins, we often take some time to reflect on the previous year and look ahead at what is to come. Suffice to say, 2016 was a momentous year for software delivery in the enterprise. DevOps adoptions matured greatly and, at conferences worldwide, industry leaders were eager to share their experience and expertise.

While DevOps was taking over, two trends focusing on ‘small’ things were making big waves: containers and microservices as a means for organizations to scale their application development and releases. While both technologies have matured considerably in the last year, they are still challenging – particularly for enterprises that need to incorporate these new trends with monolithic applications, traditional releases, VMs, etc. While it’s not a silver bullet that’s suitable for every use case, we’re increasingly seeing more enterprises exploring Docker and microservices for their needs, and I’m sure these trends will continue to lead positive advancements in modern application delivery for the coming year (and beyond).

Continue reading for my insights on some of the opportunities and challenges surrounding these two major trends, as well as a few predictions for what’s to come in 2017.

2016 Trends – Looking Back

DevOps has matured:

DevOps experienced considerable maturation and advancement in 2016. From its grass roots beginning with small-teams, mainly for green-field applications or startup companies, DevOps has matured, where complex enterprises – not just the unicorns – are now often on the forefront of DevOps innovation. DevOps practices have expanded beyond just web apps, to be used for your database deployments, embedded devices, and even mainframes.

Microservices and Containers go hand in hand:

Microservices are an attractive DevOps pattern because of their enablement of speed to market. With each microservice being developed, deployed and run independently (often using different languages, technology stacks, and tools), microservices allow organizations to “divide and conquer”, and scale teams and applications more efficiently. When the pipeline is not locked into a monolithic configuration – of either toolset, component dependencies, release processes or infrastructure, there is a unique ability to better scale development and operations. It also helps organizations easily determine what services don’t need scaling to optimize resource utilization.

However, we have seen in 2016 that adopting containers and microservices can be challenging. What’s starting to happen, and what some people are starting to realize the hard way, is that you’re going to have a really hard time doing containers and microservices well with bad architecture. Whether that is the architecture for your application, services, infrastructure and delivery pipeline – architecture matters greatly in your ability to do microservices well and take advantages of the benefit they, and containers, can offer. Microservices are not for everyone, and if you’re looking for microservices as a way to “uncomplicate” your life – you certainly won’t find that if you haven’t resolved CI, automated testing, monitoring, high availability or other major prerequisites, before. If you’re struggling in one area, it is highly recommended that you address those issues first and then you can decide if microservices will be an asset to your organization.

Legacy applications are another challenging factor when we look at the surge of demand for microservices over the past year. Do you decompose your application? Or parts of it? Do you build API around it? Do you completely re-write it? We’ve learned the value of a phased approach when re-architecting a legacy application or dealing with an organization that has established processes and requirements for ensuring security and regulatory compliance. In these instances, teams should consider starting with a monolithic application where practical and then gradually pull functions into separate services.

Microservices and containers will continue to be major players in 2017, and gradually some of the kinks and challenges associated with them will be ironed out as the community as a whole becomes more proficient with them, and tools and patterns are introduced to accelerate the adoption and large-scale operations of these technologies.

What’s Ahead in 2017  

DevOps failures will come to the forefront

While the hype around embarking on a DevOps transformation is credible and the ROI and transformational benefits of a DevOps transformation have been established, there has been a silent backlash that will make itself known in 2017. This will be the year we start to hear some of the failures – which is good because we learn from them. There will be quite a few people who say, “Yeah, we tried DevOps. It doesn’t work. It’s all hype. It only works for the unicorns. It only works for new software. It only, it only, it only…” This type of doubt is bound to happen when any new process or framework is introduced into legacy environments and cultures, and 2017 is the time for the doubt around DevOps to rise. Every DevOps adoption story is unique, and every journey is one of trial and error, and the road towards continuous improvement. We know the road is challenging, frustrating at times, and certainly it’s not all-roses. But the rewards are immense. As the community shares patterns and learning for what works, we should also be more forthcoming about sharing our failures, setbacks and wrong turns. So we can all learn, and become better at realizing software. Together.

Importance of Microservices Adoption

2017 looks to be a year where microservices and containers will continue to rise, and get pushed even further into the limelight as companies continue to invest in software-driven innovation and technology. It will be critically important for companies to have a solid architecture in place and understand how to approach and scale them effectively.

DevOps’ Impact on Financial Services

Furthermore, we will start to see some interesting disruption in FinServ. Interestingly, financial institutions have often seen technology as their own personal differentiator. However, more often than not the problem these organizations have is with supporting the right culture to enable a successful DevOps implementation. As veteran companies work on instilling the right mindset to enable faster releases in such a heavily regulated industry we could also see disruption in the space, where new financial technology, new online banks, new companies, etc. are taking up more and more market share.

Secure DevOps

With the proliferation of IoT and our always-connected world, and with the growing cyber security threats, we are going to see more security verification and more built-in compliance validation checks happening earlier in the lifecycle that are fully integrated with the development process.

Final Thoughts

All in all, what all teams and organizations need to do is keep their eyes on the ball. We practice DevOps to bring more value to our customers and employees, delight our users, and make things better, safer and faster. Keep that in mind – because if what you’re doing on any given day isn’t moving that ball forward, what was the point? Focus on that for the new year!

Behaviorally Speaking: DevOps Development

+
-
0

Behaviorally Speaking: DevOps Development
by Bob Aiello

DevOps puts the focus on developing the same build, package and release procedures to support deployment throughout the entire application lifecycle. This is a big change for many companies where developers traditionally did their own deployments and operations joined the party late in the process, usually when the application was being prepared to be promoted to production. In fact, I have seen organizations who put a tremendous amount of work into their production deployments, but failed miserably simply because they started too late in the development lifecycle. DevOps puts the focus on creating an automated application lifecycle management (ALM) to support development test, integration, QA, UAT and Production. But how exactly do you develop DevOps itself and how do you know when you have achieved success?

DevOps is not new; like many other Agile and Lean practices, DevOps makes use of principles that have been around for a long time. But, also like Agile and Lean, DevOps clarifies and highlights industry best practices in a way that is particularly compelling. Traditionally, developers have demanded free reign in being able to build, package and deploy their own work. As a group, developers, by and large, are known to be smart and hardworking folks. Unfortunately, seasoned professionals also know that many IT problems and systems outages are caused by a smart person accidentally missing an essential step. Creating repeatable processes and ensuring uninterrupted services is primaily about creating procedures that are simple, clear and reliable. DevOps teaches us that we need to begin this journey early in the process. Instead of just automating the build and deploy to QA, UAT and Production – we shift our focus upstream and begin automating the processes for development. too.

Continuous integration has been made popular by industry expert Martin Fowler who strongly advocated deploying to a test environment even for a development build. Creating seamless and reliable fully-automated deployment procedures is not an easy task. There are many dependencies that are often difficult to understand – much less automate. Developers spend their time learning new technologies and building their technical knowledge, skills and abilities. DevOps encourages improved communication between Development, QA and Operations. The most important part of this effort is to transfer knowledge earlier in the process – reducing risk by creating a learning organization. DevOps helps to improve both QA’s and Ops focus by enabling them each to get involved early in the development cycle and also by creating automated procedures to reliably build, package and deploy the application. If you want to be successful, then you should also strive to be agile.

DevOps was made popular by a number of agile technology experts, including those involved with what has become known as agile systems administration and agile operations. It is essential to remember that the journey to Devops must also follow agile principles. This means that creating your automated build, package and deployment procedures should be handled in an agile iterative way. This has been exactly how I have always handled this effort. Usually, I get called in to deal with a failed application build and release process. I often have to start by performing many tasks manually. Over time, I am able to automate each of the tasks, but this is an iterative effort with many decisions made at the last responsible moment. Source code management is also an essential starting point.

Developers need to be trained to successfully use version control tools to secure the source code and reliably create milestones – including version labels or tags. Just as DevOps starts at the beginning of the lifecycle – DevOps also needs to focus on the seminal competencies of excellent source code management. Automated application build is next, with each configuration item embedding a unique and immutable version ID. Release packages are created with embedded manifests containing a complete list of every included configuration item, whether it be a derived binary, text based configuration file or a word document containing essential release notes. Good release management means that you can identify all of the code that is about to be deployed and also provide a procedure to verify that the correct code was in fact deployed (known as a physical configuration audit). It is equally essential to provide a mechanism to ensure that a release package has not been modified by an unauthorized individual.

What makes these practices “DevOps” is the focus on developing these procedures from the beginning of the lifecycle and taking an agile iterative approach as they are developed. There is also an intentional effort to share knowledge equally between development, QA and operations. Development often knows the technology best, but operations understands the real world challenges that will result in developers being roused from their slumber in the middle of the night. QA contributes by developing the procedures to ensure that bugs are never found a second time in any release.

Devops is all about the synergy of productivity and quality with the real world focus on sharing knowledge and building competencies!

1234...11Page 3 of 11