I have recently pushed out a small update to my Profanity Detector C# / .NET Library. There are no functionality changes, but I renamed some of the functionality in the library. A central feature was the ability to add words to a ‘white-list’ if you want them excluded from the main list of profanities. With the recent Black Lives Matters protests, there has been a push to remove any language from software that could cause any racial offence.
The term white-listing and black-listing is a common idiom used in software development, but the naming could be better, such as block-lists and allow-lists. In the profanity detector library, I have renamed the ‘white-list’ feature and associated documentation to ‘Allow Lists’. Nothing has functionally changed, but if you update to the latest NuGet package, you may need to update any class name references from WhiteList to AlowList.
I have also updated the branch in GitHub that the code lived in from ‘master’ to ‘main’ as I discussed in my tutorial on branch renaming in GitHub. Thank you to all the people who use the Profanity Detector in their systems. I hope you continue to get good use from the library.
Before Christmas a good friend of mind, Rachel, asked me if I could help her set up a website for a business she was starting selling antique silverware (Argent Antiques) . Rachel needed the site to offer a standard set of e-commerce features where she could add and amend products and content on the site.
The software developer in me immediately starting thinking about how I could build her a solution. I was thinking that I could develop the site using ASP.NET MVC 5, Twitter Boostrap, Braintree payments integration etc. then then I thought, STOP, what are you doing? This isn’t a fun software development project, and Rachel certainly didn’t want to wait 3-6 months for me to lovingly hand craft the perfect e-commerce platform. She wanted a site up and running that fit her requirements in as short space of time as possible. After all, a site that isn’t live, is a site that isn’t generating revenue.
I took a step back and started looking at the different options available to me, these included :
Develop a custom site in ASP.NET MVC.
Use a CMS system like WordPress, Joomla.
Use a SaaS (Software as a Service) eCommerce solution like Shopify or Wix
The option that I selected is based heavily on the circumstances and restraints that I was working within. For example, I was asked to produce this site as a favour to a friend, which meant I was going to be working on it in my spare time. I have a full time career, plus I am a Dad to 2 kids, so spending a lot of time on a solution would be an overall constraint. Plus from Rachel’s perspective, she wanted to get a version 1 website out there as soon as possible. The screen shots of the site in this article are our current version 1.0, quick to market version of the site that is fully functioning.
Here are some of the Pros and Cons I used as part of the selection process.
Working in a global retail environment poses some interesting availability challenges when you have physical Bricks and Mortar stores. I have been thinking about the problem of high availability in this environment for a little while now due to a project I am involved with to harmonise the retail systems used between global groups. It is quite a common problem for an organisation that has grown through acquisition that you have different systems used in different business units, but after a while it makes sense to try and go with a common platform.
This article talks about how this architecture could look and how you can support the staggered roll-out of new Point of Sale features to the store whilst still maintaining high availability.
Before I talk about the architecture, I want to cover a scenario first of the end state. Imagine there is a global retail company based in both North America and Europe. Both territories have around 1000 physical bricks and mortar stores. These stores each have a number of tills (cash registers for my American friends). There could be between 2 and 5 tills per store depending on its size. Each till communicates with systems hosted at a centralised location. These systems consist of web services, caching servers and databases. This has been illustrated in the diagram below.
This diagram shows 2 geographic regions. Each Region contains a head office network infrastructure and a store network infrastructure. Both of the global regions are completely separate from each other. There are no shared resources between the two.
Since I wrote my article recently about Google’s 9 principles of innovation a reader over on Reddit pointed me to a resource on some more good design principles. These are the 10 Principles for Good Design by a German industrial designer Dieter Rams.
Dieter Rams introduced the idea of sustainable development and of obsolescence being a crime in design in the 1970s. Accordingly he asked himself the question: is my design good design? The answer formed his now celebrated ten principles.
Whilst Dieter was an industrial and product designer, his principles can fit anywhere where good design comes into play. In the rest of this article, I will explain what the 10 principles are, and how I think they fit into software development.
The principles in this article are very useful for software developers and designers, but this is also very relevant for technical leaders. As a leader it is good to try and push your teams to make sure they are thinking about the end user. Traditionally, software developers make lousy designers (not all of them before I start a flame war), but aesthetic design, generally, isn’t something that comes naturally to programmers. Therefore having principles like these is great for giving you pause to reflect on how your system / application affects your end users.
The different product images below are examples of products designed by Dieter. Those of you old enough may recognize a few!!
In this article I want to discuss a web services architecture that I have worked with over the last 7 years in a couple of different companies. The architecture I am discussing composes web services into layers, but there is much more to it than that. Nothing in this article is rocket science to an experienced services developer, but it is a way of working that has proven very reliable and easy to understand for developers that gets them working in a consistent way.
The diagram above shows what you will typically be used to working with in an enterprise. It is a standard 3 tiered architecture comprising of the following layers:
Presentation Layer: This contains any views that your users will interact with like, web sites, smart phone applications, desktop clients, etc.
Business Logic Layer: This contains the under lying logic for your application which may be comprised of web services, REST Services, windows services etc. The presentation layer shouldn’t contain any business logic and therefore should be quite dumb to the inner workings of your system, this is what the Business Logic Layer is for.
Data Layer: This contains your data storage. This typically is an SQL database (SQL Server, Oracle, and MySQL) or one of the newer blend of NoSql Databases (RavenDB, MongoDB, Cassandra). Typically your data layer may expose stored procedures to the Business Logic Layer, or your business logic may connect via an Object relational mapper such as Enterprise Library, NHibernate or SubSonic.
This article is going to concern itself predominantly with the Web Services part of the Business Logic Layer.
In this article I want to talk about an architectural pattern called Command Query Responsibility Segregation (CQRS). I will use an example to illustrate this pattern based on a recent problem we had at the company I work for and how we solved that problem. The system in question is an automated payment processor and runs on a schedule through the night and collects payments from our customers. The original CQRS pattern was written about by Martin Fowler over on his blog.
The concepts behind CQRS are very simple, but they give you some powerful advantages. What CQRS essentially does is get you to use a different model to update information (COMMAND) than the mode you use to read information (QUERY). If you think of a typical layered application you may have a client, a user interface or a background process that communicates with a business logic layer that may be comprised of a static set of libraries, Web Services or REST based services.
This business logic layer acts as a model into your domain that lets you perform operations against that model. These operations generally allow you to create, retrieve, update, or delete data associated with the model. For most scenarios this age old way of working is fine and that is certainly true nowadays. This way of working can cause you problems for systems that need to have high performance and be more scalable.
To illustrate this pattern I will discuss a payment system developed by 2 engineers on my team, Graham Johnson and Hugh Hulme. For the purposes of this article we will call it the Automated Payment System (APS). The actual system itself had a different name but APS will do for the purpose of this article. The system would run through the night and collect payments that where due from our customers. The initial system was completely synchronous and would execute in the following steps:
The system monitor discussed previously will collect a lot of data from different systems. The amount of information that is collected can be vast and in its raw form may not be that useful except to people who really understand what the data represents. The key thing with a dashboard is to break this information down for different users. This may be a complete view of the data, or just a subset for different purposes, so you have to think about the context in which the data is going to be provided and the intended audience.
For me, a key requirement of the dashboard I built was to make the data from the monitor available to my development and operations team in an easy to view format. This dashboard contains more technical information including exceptions as that is more appropriate for the intended audience. You may need to provide another dashboard to key business stakeholders. The level of information you provide here would most likely be different and contain more business level data.
Your dashboard may be viewed in different places. This might include a large TV attached to the wall, projected against a wall, or the dashboard might be run on a person’s desktop computer. If you are displaying the dashboard in a place where people cannot interact with it, i.e. on a large TV Screen, then you want to make sure the dashboard automatically refreshes itself. I will cover this more later on in the article.
Types of Dashboard
You really are spoilt for choice these days when it comes writing a dashboard. You can keep the display simple or you can really embellish the presentation. As I said earlier, it depends on your target audience. I personally find that keeping the display simple is the best thing to do. I have seen some dashboards that go overboard with dials, speedometers and other fancy graphics, and these can confuse the information you are trying to communicate if you are not careful.
You can develop your dashboard as a native application. This is what I did with my own solution for the company I work for. I work at a company that is based around the Microsoft stack, so this really gave me 2 choices as a .NET developer; Winforms or WPF. As I needed to get the dashboard up and running quickly I used Winforms, mainly because I am very familiar with this particular technology. WPF would also have been a very good choice.
My dashboard was aimed squarely at a technical audience, so the level of detail could be higher. The context of the dashboard was to allow developers spot any problems in our critical systems. I decided to keep the screen design very simple. I used a tabbed control, where each tab represented one sensor from the xml stream written out by the monitoring system.
Recently, I have been doing lots of recruitment for .NET consultants. Each of the CV’s we receive all stress that they are experienced in working in agile development shops. This is great. We are an agile development company so these people sounds like a great fit.
So we get them into an interview and ask them, ‘What does agile mean?’, ‘How do you know if your team is truly agile?’ It’s at that point we get the standard list of responses:
We do Test Driven Development.
We pair program.
We use continuous integration.
We use SCRUM, KanBan, XP etc.
Use work in iterations.
We use story points.
We calculate team velocities.
These answers are all well and good, but they don’t describe what an agile team is. These are all just facilitators to being agile. What’s even worse is that these interviewees seem to have not heard of the agile manifesto.
This article will talk about the deployment of your system monitor to start monitoring production systems. A lot of what I discuss here is from experience, especially as I work in a regulated industry (Financial Services), which threw up a few interesting challenges on how I could achieve rapid deployments and application feedback without breaking any rules.
Deploying into a Production Environment
The goal of your monitoring tool is to regularly check over your critical production systems, so that means you need to deploy into production. If you work in a smaller company that doesn’t have a long and complex deployment process (By ‘complex deployment process’, I mean a process heavily bogged down with paper work a company processes), then this should be straight forward.
When developing monitors you want to be able to iterate quickly, and deploy early and often. If you work in an environment where you can do this and release into production quickly, then great. You need to do, just that. Every time you get a piece of functionality ready for prime time, release it. You just need to make sure that the server you deploy the monitor too can access any log files or databases to gather its information.
The diagram above shows what a typical deployment might look like. You have your monitoring server setup containing the monitor application and the task scheduler. This application then processes logs contained on each target system. The example above has web services, a website, and payment gateway and payment processors. This shows you your most basic type of deployment. This is great if you can do this type of deployment quickly and often.
This is the second article in a series on application systems monitoring for software developers. In the first article I discussed the basic idea and concepts around systems monitoring. In this second article I will go over the basic architecture of how an application monitor can work. The system will start out simple to begin with and be extended over time. This iterative approach to building up the design mirrors how I implemented this type of system for my current employer. In true agile style I wanted to get something basic working as quickly as possible so we could start getting the benefit from it early on. Once the system was out there and working, I then built upon the basic idea with new features.
First of all let’s look at the basic use case diagram above. This diagram shows 2 subsystems, the monitor sensors and the monitor dashboard. This article will focus on the first subsystem, Monitor Sensors. At this stage in the systems evolution the main actors here are Developers and Technical Support.
Roles may differ in your own organisation, but because I work in Financial Services we have to have a clear separation of concerns between the development infrastructure and the production infrastructure. This means developers can not directly deploy too or modify a production environment. This is for good reason; we are very good at breaking stuff!! This is why we and many other large organisations have a separate technical services teams who maintain the production environment.
That means both developers and technical support have a vested interest in the results of the monitor. Just because developers can’t make changes to a production environment, there is no reason why they can’t see the telemetry data collected by the monitoring systems. In fact it would be a very bad idea if they couldn’t. By seeing how the production systems are running directly from a snapshot data stream, developers can get a good impression of how their systems perform in a real production environment. This may well help to influence future design decisions when the systems are extended. This also gives developers a realistic impression of what volumes their systems cope with in real life. As we all know a development test environment, no matter how hard we try, never matches what we see in a real production scenario with real users and data.