Archive 7 min read

Working With Data

There's a lack of easy-to-use tools for "average" users to analyze their data. In today's world, that means something.

Working With Data

Watch this episode on YouTube.

Reasonably Accurate 馃馃 Transcript

Good morning. How's everyone doing today? Um I hope you guys are having a good day. Um Yesterday, I talked a bit more about Facebook. Um and I was working on that post about um what's included in the data download. Um kept um kind of plugging away on that tool a bit yesterday.

Um Have some more stuff. Uh The post isn't finished yet. As always, things take a little longer than expected. Um But I want to show you a couple of things actually um today on screen um because I think that would be important but what really kind of um delayed getting the post ready.

Um Let me just kill the audio in the background here or really kind of got the post um or delayed. The post was I was really scrambling around to try to find a tool to properly illustrate the map. Um So really, um you know, visualizing geospatial data fancy way for making maps or saying make a maps.

Um I there's a ton of different tools out there, but they all kind of suck. Um And I think that's a big problem because as we're having more and more data um and access to that data. How do you make it useful?

Right? You need to be able to process it automatically, you need to be able to script that, but you also need some handy tools where you can just play around with it. And uh there's a few commercial offerings out there. Um I was running through all that tried through a bunch of open source ones and I got some decent results.

Um yesterday on the show here, I had shown you um the web view um which is great for interactive, um you know, for showing it on a page, but I don't really need this to be interactive data. What I had in my mind was actually just a pretty straightforward movie uh of, you know, showing a map and then data populating in with the date um over time.

All right. So show a year's worth of Facebook activity um and show that popping up on the map as different things were logged in the data download. Um shouldn't be really hard to do. Apparently it's kind of tricky. Um There's a number of things that need to go in there.

Um Of course, you know, um I do a lot of work in Python, which is programming language. Um And there's a great mapping tool kit in there, but like most open source projects, this giant pain in the, you know, what to try to get installed and set up.

So, not really usable for a script that I'm going to release out into the public. Um Unless I want to provide a whole bunch of extra instructions which I'm not super interested in. I don't think anybody would be super interested in trying either.

So right now, the tool just dumps out our data in a couple of different formats. Um So like CS vs or comma separating values or some JSON make it usable in your favorite tool. Um So I'll wrap that up and get that out as quick as I can.

But um on the side of data visualization, things like that. Um this afternoon, I'm going to a um event uh mini conference here in town for the government of Canada looking at um future technologies and foresight, um applying data from foresight um into government policies and things like that.

So it should be really interesting. I was invited as an external participant um which is a nice little honor and I really appreciate that. Um It should be a fascinating afternoon of discussions um really looking forward to it, but of course, it cuts into time of getting stuff like this ready.

So um little bit of a challenge there. But uh as always, you know, as you have seen uh following along on mornings with Mark here, things rarely plan uh work out as planned for my day. It's a miracle that anything gets done.

But I wanted to show you guys a little bit of my screen here. Um And I'm just gonna rotate, I'm gonna kill my cam for a sec because this will look better. Um So on the uh side of visualization. So let me just double check this.

Yeah, perfect. OK. Um So this was the tool that I forgot about, which is ridiculous because uh uh as many of, you know, I'm an Aws community hero. This is Amazon clicks site. Um And it was really, really simple to get this up and running um like five seconds kind of easy.

Um And what I've got here, um and I won't go into how or how I set that up, but pretty much, you know, I gave it a common separated value file and then said, display it in a map and now I've got a different color dot um based on the date.

Um And then the size is the number of things that occurred at that location. Um And you can see, you know, here's my activity for 27 to 2014 through to 2018. The activity really only includes um photos from earlier than 2017.

So, um I can easily set up a filter um and not allow uh so I can add a filter and say, you know, hey, don't display anything from 2014, just give me 2017. Um But you can see this is all in my personal Facebook data dump.

So you can see, I do get around the world quite a bit um down through Texas and the, you know, Canada in the United States um in general, but then through Tokyo quite a bit um down to Australia once and then into the EU as well.

Um But yeah, I totally forgot about Amazon Quick site and it was a, it's a fantastic tool if I want to look at this in a completely different way. Um I just click on a different chart and it will show me a different progression and I can just play around with the different um visualization types um and add and replace whatever I need.

And uh there we go and I go back to a map. Um Very cool tool. I totally forgot about it, which is a all sorts of irony given my position as a community hero. Um But I'll be posting uh this map display out um on the blog post, Checkmark and dot C A for that um later today or tomorrow.

Um Same with Twitter. I'll post a link there. The other thing I wanted to talk about was this interesting little parser bug that somebody had posted about. Um Some people were starting to freak out a little bit about it um As far as the I OS camera.

So your iphone ipad camera, if you take a photo of a QR code, this is a QR code here, this ugly blob. They're not extremely popular here in North America. But um in Asia Pacific, you see them quite a bit essentially what it is is if you scan this with your smartphone camera, it will display a link.

Um So you can see here this little gif uh of this happening. So he scan it with his camera and voila it takes you to a website. So he's looking at it here. Click on the notification. Yeah, go to that and off it takes you to the website very easy way to get complicated URL S into mobile phones.

But what this vulnerability is or what this issue is is essentially there's a way to format your QR code, which is just an encoded URL um to take you somewhere else. So in this case, it's saying go to Facebook but then really go to this guy's website and so the notification says Facebook, but it then takes you somewhere else.

This is not a massive security issue. It's a quality control issue in the parser. And they did not write this strongly enough, but it reminds everybody yet again um that you need to very, very much um be aware uh of when you're clicking on notifications, when you're taking an input from other things when you're visiting random URL S, make sure that you're actually getting to the site that you think you're getting to.

Um So a little interesting thing there and I saw that picking up theme. So I wanted to talk about it a bit because it kind of goes to a larger theme of, we are expected to kind of click on things um and respond to all these notifications.

And we very come, we become very habitual to it and we don't really think about it too much, just tap on it and go um very trivial example here um of, of a malicious notification. Um But we are starting to see those pop up more and more and again, a good reminder where um you know, make sure the phone and the tools that you are using are serving you and not serving someone else's purpose.

And I know that is sort of a vague platitude but it is what it is. Um that's kind of a seed of an idea that I'm sure will pop up into something more. Um But my uh short answer right now, my, my immediate thing is um looking at that Facebook data, I really like to get that out because you're seeing um Facebook executives start to testify in various places.

Um And that is gaining steam as a um Chris Vickery over at up guard found a new data trove from a subcontractor to Cambridge Analytica that exposes a whole bunch more stuff. And this is going to be another go round of data analytics and tracking, which I think is a critical issue.

So um that is the morning for me and then this afternoon that great event for the GOC uh other than that. I hope you, you are having a great day. As always. There we go. Uh marknca hit me up here in the comments online.

Always happy to chat. I know I've got a few replies. Outstanding right now. I will get to them. I do get to them. Um Sometimes it just takes a little while so bear with me. I hope you guys have a great day and we will talk to you.

Read next