Under the Bonnet - Volume 1 | Issue 1 - 4 May 2016
Welcome to the first issue of Under the Bonnet, the briefing document series issued by HATDeX (HAT Data Exchange) to members of the HAT network. This document series will provide briefings on different aspects of the HAT; its tech, business, markets as well as product updates.
In this issue:
Page 1 - Delivering the HAT: We explain the technical solution of how we are rolling out HATs and how organisations can get involved in offering HATs to their customers
Page 4 - Beta Services coming in July: We explain the two important services that are coming out with your HAT in July
Page 5 - Sneak Peek: A preview of your HAT profile page
Delivering the HAT
As the operator of the HAT ecosystem, we have always maintained that the HAT, as an open-sourced community-based network, is open to anyone taking the HAT schema, logic and APIs to become a HAT Platform Provider (HPP) on their own hardware or cloud infrastructure to build and commercialise for free. This is available at https://github.com/Hub-of-all-Things/HAT2.0. This is our primary provisioning model: supporting such organisations to develop and provide interoperable personal data exchange platforms admitted as HATs to the HAT ecosystem through HATDeX offering a process of certification to deliver assurance that the HATs operate in secure, private and trusted environments.
While organisations can take the HAT database schema, logic and APIs and become a HPP, we have however, been asked to build the services to deploy HATs. This follows our successful Indiegogo campaign. Hence, the HATDeX tech team has been busy working on getting a particular HPP solution for Beta HATs ready for rollout in July, so providing our secondary provisioning model.
Well, it took us a bit of time (took us ages to draw this!!!), but we’re happy to now be able to illustrate what will be coming out with the HAT Beta release (see diagram below).
First of all, the HAT is at the centre of the diagram. This is the dedicated HAT Database sitting in a Docker Container (https://www.docker.com/what-docker) that every HAT user will have. Each database contains a data schema, allowing for the storing of an individual’s data from any source without losing the structure specific to the source, while at the same time enabling the individual to relate their data to the context of their personal life and provide a common semantic structure for third parties to use such data.
We have always emphasised the need for our solution to keep “personal data personal”. Namely, a user’s HAT data should not be stored in an account-based system, where each user is a (series of) record in a database hosted by a service provider, but ideally stored in a (virtual) private server that is accessible only to the user. Our solution for this is the use of virtual machines (VM). However, it is very expensive to offer each user a full VM just to run their HAT and related light services, when a VM is designed to support various operating systems with much broader application scopes.
Our solution for the HAT rollout is to deploy Containers, and in this case Docker Containers. While Containers are not fully-scoped servers, they wrap up a complete system that contains everything it needs to run: code, runtime, system tools and system libraries. With your HAT in a Container, it will be lightweight and will always run on a wide range of infrastructures – meaning it can run on any computer, on any infrastructure and in any Cloud. Each Container runs a complete and independent HAT, and the only way to communicate with a HAT is via designated APIs. The Container-based solution also provides an added layer of protection by localising impact of any security issues and eliminating the possibility of system administrators enabling unauthorised access to the data of large numbers of users.
HAT APIs were developed to exercise user-managed control of personal data. REST APIs for the HAT are the only mechanism through which a HAT may be accessed by the web, mobile and other clients that interact with the HAT, allowing the user to control their data and applications to benefit from it. API documentation can be found at http://hub-of-all-things.github.io/doc. We implement End-to-End encryption for HAT data via APIs – encryption of data at rest, encryption of databases and SSL-encrypted data transport.
Hosting HATs on Amazon Web Services (AWS)
To make it possible for corporations or even advanced users to host HATs and/or offer HATs to their customers, we will run them on Amazon Web Services (AWS). Each HAT will run in a Container and will talk to a separate database server to which no other service has access. Each HAT will communicate with the outside world via APIs only, and can be accessed only by its owner/user and authorised applications from the outside world. The HAT’s database will run on HAT Database Servers, which will also operate as isolated Containers across a number of Elastic Compute Cloud EC2 instances for reliability. HAT data is Containerised personal data storage per user for the proposed solution has never been scaled before. Hence, the HATDeX team has developed the unique Milliner Service to manage automated deployment, operations and scaling of Containerised HATs. Learning from the best, we created the Milliner as a layer on top of Google’s open-source Kubernetes, leveraging its true freedom to work on various Cloud infrastructures. The Milliner orchestrates all HATs and their databases, including creating / deleting HATs, backing them up, putting them to sleep and waking them up when needed, as well as elastically adjusting the resource required.
HxPP: The adoption of Kubernetes and Docker Containers means that instead of being a HAT Platform Provider (HPP) provisioning everything themselves and writing code from scratch around the HAT database, companies can license the Milliner service from HATDex and be a HATDeX Platform Provider (HxPP) . They can then host HATs pretty much anywhere Docker Containers and Kubernetes can run (we have even seen people doing so on a Raspberry Pi), fitting their needs and on their preferred infrastructure. By licensing the Milliner Service from HATDeX, HxPP will also have access to future backend services offered through the Milliner.
HSP: For organisations that do not have the infrastructure capability to become a HxPP, but with some software capability, we have developed Milliner APIs to enable them to become HAT Service Providers (HSPs). This means that HSPs can offer HATs to their customers by using Milliner APIs on a Platform-as-a-service basis. HSPs can deploy any applications (web / mobile) to provision, run and manage their customers’ HATs in a Cloud environment provided by HATDeX (we currently use AWS). Being a HSP is useful for corporations who wish to reduce personal data storage and support costs as well as to mitigate reputational risk from privacy and security management of personal data, outsourcing hardware and software maintenance of HATs to HATDeX. Our HSP provisioning model is useful for software developers, SMEs and manufacturers of IoT products with their own software capability around personal data.
HoD: HAT-on-demand (HoD) is a provisioning model for organisations or individuals with no infrastructure and no software capability. It is a step up from the HSP provisioning model in that the HAT, as well as the Rumpel HyperData browser, is provided on a software-as-a-service basis. This provisioning model is provided to organisations with a full white-labelled solution. The HoD provisioning model is useful for academic projects, SMEs and organisations without any software or infrastructure capability for personal data.
Our HATDeX provisioning models ensure that organisations can be involved in offering HATs to their customers from a small-scaled pilot to fully-scaled HAT hosting on the chosen infrastructure. If your organisation is interested to be involved, please email us at email@example.com.
Interacting with your HAT - Beta Services coming in July
Hyperdata Browser: Rumpel
Rumpel is the world’s first HyperData Browser for HAT users to view their own personal data. To understand what Rumpel really is, remember that the full collection of your personal data is very different from the data that you would give to an organisation with whom you interact; the latter is generally “siloed” (data of one type such as identity or card numbers, or your email or photos), whereas the totality of your personal data is a combination of information from many sources. This data is also dynamic and will include data from public and private sources, including your own records – diary, location, weather, pollution at your location, your social media postings and postings by others, photos, etc. However, we have never seen all of this data in one place, and therefore cannot appreciate its value to us.
Rumpel fulfils this important function for personal data, making it a truly personal HyperData Browser with data visualisations and allowing individuals to organise it into their own bundles and collections. It can be always close to the individual, being available on both the web and mobile, so Rumpel benefits the individual, providing personal data in context and on demand. And in so doing, we would like Rumpel to ‘spin straw to gold’ – to make individuals realise that their personal data has immense value.
Now that the individual is sitting on a goldmine, s/he must also be able to exchange some bits of it! A Data exchange app comes within Rumpel, offering what others wish to trade for your data and allowing marketers to buy it from you with vouchers, discount codes, freebies or even cash. Even then, individuals get to control what part of the data they allow access to without a chance of anyone seeing anything else.
MarketSquare – Marketplace for HAT Data and community building
You can also be sociable with your personal data, as we are building the MarketSquare for you and your data. In the MarketSquare, HAT users can get together to talk about the best way to share our data; hang out and chat with people like ourselves; compare the market and see what cool data offers are out there; compete with friends on who’s earned the most from personal data; exchange usage data and make collaborative consumption possible (i.e. offer your vacuum cleaner available to your neighbours when you are not using it); create guilds and groups or rally your HAT friends to your own social movement to claim personal data back from corporates!
Sneak Peek into: Your HAT profile page
Your HAT address is where and how you sign in to your HAT. In July, this would be name.hubofallthings.net. As you can probably see, it looks like a URL, which it is. So if you actually type name.hubofallthings.net on your web browser, it will take you to your public-facing HAT page.
For the beta version in July, this HAT page is just a placeholder page with limited customisability. Users will tick the attributes on their HAT profiles (accessible on Rumpel) that they are willing to make public, or they can decide to keep their HAT page fully private. The two images on the left show you how the front page of a HAT would look like.
HAT users would eventually be able to replace this page with any information they wish to broadcast (eg a personal website, a blog page, a storefront, a video channel, art, video or media created by the user etc). The information on these pages are what users have designated to be public, but they form a part of their HAT data which means they can be indexable, searchable and even shared!
Customisation of these pages is in the development plan. If you are a developer, you can help expedite this by joining the open source HAT community and help us build on the code.