Guide: Connect Looker to Druid and explore your data in real time

In a previous blog post we highlighted the importance of leveraging data interoperability for data visualization. We recognize that interoperability is key and we strive to support your team with whatever data visualization tool you like best. Here, we present a step by step guide on how to leverage Looker’s modeling and ad-hoc analytics with the performance of Druid to generate an operational analytics experience that is fast and interactive. 

If you use Tableau, check out our detailed guide on how to connect Tableau to Druid for sub-second dashboards. 

‍

Druid is fast - no caching necessary

‍

Many analytics tools feel sluggish when connected to a data warehouse. If you have highly dimensional data, complex aggregations, or billions of rows, these can contribute to an analytics experience that is so slow that you are unable to iterate effectively. Real-time and intuitive dashboards are key when trying to explore and make decisions and a slow experience will hinder your team’s desire and ability to dive into the data. 

To improve performance, Looker provides powerful caching features as well as persistent derived tables, which store results that can simplify your queries. These features provide speed if your data is static. However, Druid is frequently employed with streamed, very recent data. If the data you want to analyze is always recent, caching is not effective.

Fortunately, caching is not necessary to achieve performance when working with Druid. If you are working with time series data, Druid’s segment architecture and powerful indexing typically provides query performance that is at least 10 times faster than your standard cloud warehouses. In addition, Rill’s pricing model encourages you to query without limit. You are charged based on the amount of data that is loaded, rather than by number of queries.  

Looker Explores - a perfect complement to Druid

‍

Looker’s “Explore” architecture, which encourages the business user to start with a dashboard and then dig deeper via ad-hoc exploration, is an excellent complement to Rill’s fully managed service for Apache Druid. At Rill, since the cost of usage is based on the amount of data loaded, you are free to query freely without worrying about running up a large bill! 

Together Looker and Druid give you interactivity and speed - the necessary ingredients for finding operational insights on the fly. If you have access to Looker, use this short guide to walk you through how to connect Looker to Rill’s public Druid cluster. Once you follow these simple instructions, you’ll be able to create Looker Explores on top of any of Rill’s public datasets. From the Looker UX, you’ll experience the quick and intuitive dashboards that you can achieve when Looker is powered by Druid.

‍

Getting started in Druid

‍

1. In order to access Rill’s public Druid cluster from either Looker or Druid, you’ll first need a login. To create your login, simply log in to Rill with any G-Suite account. You’ll find yourself in the Rill Cloud Console (RCC) - this is your interface to the Rill service. It bundles Druid as a secure, fully managed service, allowing you to load and query data in a managed Druid cluster.
2. Once you are logged in, you’ll be part of a workspace called “Public”, which contains a number of preloaded datasets. You’ll be viewing a screen that looks like this:

3. If you’d like to explore these datasets from within RCC, click on Druid Console. This will open the Apache Druid console in a new tab. Note that the “look” changes because you are no longer looking at RCC; rather, you are now viewing the open source version of Apache Druid, which Rill layers with security and scalability. 
4. Click on a dataset such as “citibike_tripdata” in the left panel to see its columns. You can type a select query directly into panel at the top, for example, “select * from citibike_tripdata”. Then type Run to run it.  
5. To prepare for connecting from an external application, you’ll need to create an API password in RCC. Return to the RCC tab and click on your name at the top of the black panel and then click Account. From Account, click on Enable to the right of API Password, create your password, then click Enable Password. You’ve just created a password that can be used from an external application to access the druid data in your workspace.
6. Click the back button in your browser to return to the RCC Datasets page.

‍

Visualizing your data in Looker

‍

Ok, so now you have some data in Druid, but what can you do with it? Let’s bring it into Looker.

1. In RCC, click on Integrations. You’ll see this:

2. Click on the copy button in the Looker section to copy the Remote Host:port to your clipboard.
3. Log in to your Looker account
4. Click on Admin -> Connections -> Add Connection
5. Specify a name for your connection
6. Set Dialect to Apache Druid 0.18+
7. Paste the Remote Host:port that’s in your clipboard into Remote Host:Port
8. Set Username to the email address that you used to login to RCC
9. Set Password to the API password that you created
10. Check the following three boxes: SSL box, SQL Runner Precache, Schema for SQL writing
11. Click “Test These Settings” to confirm that you’ve filled the form correctly. If you filled it out correctly, you will see the message “Can connect” below the button. 
12. Click “Save Connection”  
13. Now create a project by clicking Develop -> Manage LookML Projects -> New LookML Project
14. Name your project
15. In the Connection field, choose the connection you created above
16. For “Build Views From”, leave “All Tables” selected to see all of the tables in your workspace
17. Click “Create Project”
18. To create a Looker Explore, go to the Looker Explore menu and choose the project that you just created. Click on the citybike_tripdata table.
19. You should now be in an Explore
20. Click on Time Date -> Year and Count and then click Run to see number of trips by year

Next steps

‍

This short guide walks you through how simple it is to connect Looker and Druid. If you’re interested in real-time, operational analytics and insights into your business with no DevOps overhead, check out our tutorial that leads you through other features and functionality provided by Rill. If you have any questions about the Looker Druid connector or the Tableau Druid connector, please reach out for more information. Our team is here to help!

‍