https://vy.github.io// Volkan Yazıcı's Soap Co. 2016-10-04T18:40:00Z Volkan Yazıcı https://vy.github.io/ tag:vy.github.io,2016-10-04://blog/post/2016/10/04/coders-at-work/ Notes on "Coders at Work" 2016-10-04T18:40:00Z 2016-10-04T18:40:00Z <p>There is nothing like thinking about work while your are on vacation. And that was indeed what I did: Reading <a href="http://www.codersatwork.com/">Coders at Work: Reflections on the Craft of Programming</a> in a tango-themed Buenos Aires trip.</p> <p>I had already met with Peter Seibel in his well-known splendid work: <a href="http://www.gigamonkeys.com/book/">Practical Common Lisp</a>. He definitely has a knack for transforming technically challenging problems into a pedagogically digestable <a href="https://en.wikipedia.org/wiki/Nootropic">nootropic</a>. This uncanny ability was placidly lurid in Coders at Work as well. Motivated by the mind provoking exquisite content, I felt an urge to keep a record of its reflections on me.</p> <h1 id="on-the-content">On the Content</h1> <p>I totally enjoyed the book and read it cover to cover. Nevertheless, I believe the following subtleties could have been addressed in a better way.</p> <ul> <li> <p>A majority of the interviewed <em>coders</em> are not actively <em>coding</em> any more. I find this a little bit contradictory with the title of the book. While the content still makes a great deal about the historical progress of programming and programmers, I find the detachment of the interviewees from the modern computing slightly unexpected.</p> </li> <li> <p>Given the back that the book examines the events dating back to more than half a century, I sometimes find myself lost in the time context. Additional footnotes to enhance these kind of ambiguities could have been useful.</p> </li> </ul> <h1 id="highligts">Highligts</h1> <p>Below I collected my personal highligts on certain statements that are shared by certain interviewees.</p> <ul> <li> <p>In general, <em>coders</em> do not practice software testing extensively. Further, I had the impression that that they do not read much programming books either. This issue sometimes acclaimed to the lack of necessary set of fundamental books at the early stages of their career.</p> </li> <li> <p>Among the entire deck, I find Joshua Bloch, Bernie Cosell, and Donald Knuth the ones with the most sensible and to-the-earth statements.</p> </li> <li> <p>A notable subset of the interviewees dragged into computers not by a deliberate decision, but by chosing a yet another career path that was available to them. (For instance, Fran Allen got a Fortran instructor position in IBM in order to finance her school loans to be able to continue her math teacher career pursuit.)</p> </li> <li> <p>None believes that reading Knuth’s <a href="https://en.wikipedia.org/wiki/The_Art_of_Computer_Programming">The Art of Computer Programming</a> is a must read for programmers, nevertheless they acknowledge that it is good to have it under your hand for reference.</p> </li> <li> <p>Except Knuth himself, nobody practices literate programming. (I am astounded to observe how Seibel is biased to ask this non-sense question which delivers no practical value at all to every single candidate.)</p> </li> <li> <p>Majority agrees that programming is a way more complicated and challenging occupation than it once used to be in the past.</p> </li> <li> <p>More than half thinks that good writing skills are a big plus (some even state necessity) for programming.</p> </li> <li> <p><code>printf</code> is the clear winner as the debugging tool of preference among interviewees.</p> </li> </ul> <h1 id="quotes">Quotes</h1> <p>Below you can find some snippets that I find worth mentioning from the book.</p> <h2 id="jamie-zawinski">Jamie Zawinski</h2> <p>I wish I would have known this when I was in high school. Could not agree more.</p> <blockquote> <p><strong>Zawinski:</strong> When you’re in high school, everyone tells you, “There’s a lot of repetitive bullshit and standardized tests; it’ll all be better once you’re in college.” And then you get to your first year of college and they’re like, “Oh, no – it gets better when you’re in grad school.” So it’s just same shit, different day – I couldn’t take it. [p5]</p> </blockquote> <p>His comments on C++, which are shared by many other interviewees throughout the book:</p> <blockquote> <p><strong>Zawinski:</strong> … when you’re programming C++ no one can ever agree on which ten percent of the language is safe to use. [p20]</p> </blockquote> <p>The sad truth about F/OSS:</p> <blockquote> <p><strong>Seibel:</strong> Isn’t it exactly this thing – someone comes along and says, “I can’t understand this stuff. I’ll just rewrite it” – that leads to the endless rewriting you bemoan in open-source development?</p> <p><strong>Zawinski:</strong> Yeah. But there’s also another aspect of that which is, efficiency aside, it’s just more fun to write your own code than to figure out someone else’s. So it is easy to understand why that happens. But the whole Linux/GNOME side of things is straddling this line between someone’s hobby and a product. Is this a research project where we’re deciding what desktops should look like and we’re experimenting? Or are we competing with Macintosh? Which is it? Hard to do both. [p23]</p> </blockquote> <h2 id="brad-fitzpatrick">Brad Fitzpatrick</h2> <p>His thoughts on finishing a project, which I sadly share as well:</p> <blockquote> <p><strong>Fitzpatrick:</strong> The projects that I never finish … it’s because I did the hard part and I learned what I wanted to learn and I never got around to doing the boring stuff. [p20]</p> </blockquote> <p>He is also poisoned by LWN, Reddit, etc.</p> <blockquote> <p><strong>Fitzpatrick:</strong> I like working alone but I just bounce all over the place when I do. On a plane I’ll bring extra laptop batteries and I have a whole development environment with local web servers and I’ll be in a web browser, testing stuff. But I’ll still be hitting new tabs, and typing “reddit” or “lwn” – sites I read. Autocomplete and hit Enter, and then – error message. I’ll do this multiple times within a minute. Holy fuck! Do I do this at work? Am I reading web site this often that I don’t even think about it? It’s scary. I had a friend, who had some iptables rule, that on connection to a certain IP address between certain hours of the day would redirect to a “You should be working” page. I haven’t got around to doing that, but I need to do something like it, probably. [p73]</p> </blockquote> <h2 id="douglas-crockford">Douglas Crockford</h2> <p>Why programming is difficult?</p> <blockquote> <p><strong>Crockford:</strong> Part of what makes programming difficult is most of the time we’re doing stuff we’ve never done before. [p110]</p> </blockquote> <p>He talks about his preferred way for interviewing job candidates, which is also shared by other coders in the book.</p> <blockquote> <p><strong>Crockford:</strong> The approach I’ve taken now is to do a code reading. I invite the candidate to bring in a piece of code he’s really proud of and walk us through it. [p129]</p> </blockquote> <h2 id="brendan-eich">Brendan Eich</h2> <p>Nothing noteworthy, you may guess why.</p> <h2 id="joshua-bloch">Joshua Bloch</h2> <p>Is Java off in the weeds?</p> <blockquote> <p><strong>Seibel:</strong> … is Java off in the weeds a little bit? Is it getting more complex faster than it’s getting better?</p> <p><strong>Bloch:</strong> That’s a very difficult question. In particular, the Java 5 changes added far more complexity than we ever intended. I had no understanding of just how much complexity generics and, in particular, wildcards were going to add to the language. I have to give credit where is due – Graham Hamilton did understand this at the time and I didn’t.</p> <p>The funny things is, he fought against it for years, trying to keep generics out of the language. But the notion of variance – the idea behind wildcards – came into fashion during the years when generics were successfully being kept out of Java. If they had gone in earlier, without variance, we might have had a simpler, more tractable language today.</p> <p>That said, there are real benefits to wildcards. There’s a fundamental impedance mismatch between subtyping and generics, and wildcards go a long way towards rectifying the mismatch. But at a significant cost in terms of complexity. THere are some people who believe that declaration-site, as opposed to use-site, variance is a better solution, but I’m not so sure.</p> <p>The jury is basically still out on anything that hasn’t been tested by a huge quantity of programmers under real-world conditions. Often languages only succeed in some niche and people say, “Oh, they’re great and it’s such a pity they didn’t become the successful language in the world.” But often there are reasons they didn’t. Hopefully some language that does use declaration-site variance, like Scala or C# 4.0, will answer this question once and for all. [p191]</p> </blockquote> <p>On “obviously no deficiencies” versus “no obvious deficiencies”:</p> <blockquote> <p><strong>Bloch:</strong> There’s a brilliant quote by Tony Hoare in his Turing Award speech about how there are two ways to design a system: “One way is to make it so simple that there are <em>obviously</em> no deficiencies and the other way is to make it is so complicated that there are no <em>obvious</em> deficiencies.”</p> <p>The paragraph that follows is equally brilliant, though it isn’t as well-known: “The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature. It also requires a willingness to accept objectives which are limited by physical, logical, and technological constraints, and to accept a compromise when conflicting objectives cannot be met. No committee will ever do this until it is too late.” [p197]</p> </blockquote> <p>Smart people and programming:</p> <blockquote> <p><strong>Seibel:</strong> Speaking of writing intricate code, I’ve noticed that people who are too smart, in a certain dimension anyway, make the worst code. Because they can actually fit the whole thing in their head they can write these great reams of spaghetti code.</p> <p><strong>Bloch:</strong> I agree with you that people who are both smart enough to cope with enormous complexity and lack empathy with the rest of use may fail prey to that. They think, “I can understand this and I can use it, so it has to be good.” [p202]</p> <p>…</p> <p>There’s this problem, which is, programming is so much of an intellectual meritocracy and often these people are the smartest people in the organization; therefore they figure they should be allowed to make all the decisions. But merely the fact that they’re the smartest people in the organization doesn’t mean they should be making all the decisions, because intelligence is not a scalar quantity; it’s a vector quantity. And if you lack empathy or emotional intelligence, then you shouldn’t be designing APIs or GUIs or languages. [p203]</p> </blockquote> <h2 id="joe-armstrong">Joe Armstrong</h2> <p>On paralyses of choice:</p> <blockquote> <p><strong>Armstrong:</strong> The funny thing is, thinking back, I don’t think all these modern gizmos actually make you any more productive. Hierarchical file systems – how do they make you more productive? Most of software development goes on in your head anyway. I think having worked with that simpler system imposes a kind of disciplined way of thinking. If you haven’t got a directory system and you have to put all the files in one directory, you have to be fairly disciplined. If you haven’t got a revision control system, you have to be fairly disciplined. Given that you apply that discipline to what you’re doing it doesn’t seem to me to be any better to have hierarchical file systems and revision control. They don’t solve the fundamental problem of solving your problem. They probably make it easier for groups of people to work together. For individuals I don’t see any difference.</p> <p>Also, I think today we’re kind of overburdened by choice. I mean, I just had Fortran. I don’t think we even had shell scripts. We just had batch files so you could run things, a compiler, and Fortran. And assembler possibly, if you really needed it. So there wasn’t this agony of choice. Being a young programmer today must be awful – you can choose 20 different programming languages, dozens of framework and operating systemsand you’re paralyzed by choice. There was no paralysis of choice then. You just start doing it because the decision as to which language and things is just made – there’s no thinking about what you should do, you just go and do it. [p210]</p> </blockquote> <h2 id="simon-peyton-jones">Simon Peyton Jones</h2> <p>Testing an API in Microsoft:</p> <blockquote> <p><strong>Peyton Jones:</strong> Well, they also do some interesting work on testing APIs. Steven Clarke and his colleagues at Redmond have made systematic attempts to watch programmers, given a new API, talk through what they’re trying to do. And they get the people who designed the API to sit behind a glass screen and watch them.</p> <p>And the guys sitting there behind the glass screen say, “No, no, don’t do that! That’s not the right way!” But it’s soundproof. That turns out often to be very instructive. They go and change their API. [p253]</p> </blockquote> <h2 id="peter-norvig">Peter Norvig</h2> <p>On the traditional master and apprentice approach:</p> <blockquote> <p><strong>Norvig:</strong> But I think part of the reasons why you had master and apprentice is because the materials were rarer. When you were doing goldsmithing, there’s only so much gold. Or when the surgeon’s operating, there’s only one heart, and so you want the best person on that and you want the other guys just helping. With coding, it’s not like that. You’ve got plenty of terminals. You’ve got plenty of keyboards. You don’t have to ration it. [p295]</p> </blockquote> <p>Why programming is not an art, but a craft:</p> <blockquote> <p><strong>Seibel:</strong> As a programmer, do you consider yourself a scientist, an engineer, an artist, or a craftsman?</p> <p><strong>Norvig:</strong> Well, I know when you compare the various titles of books and so on, I always thought the “craft” was the right answer. So I thought art was a little pretentious because the purpose of art is to be beautiful or to have an emotional contact or emotional impact, and I don’t feel like that’s anything that I try to do. Certainly I want programs to be pretty in some ways, and sometimes I feel like I spend too much time doing that. I’ve been in a position where I’ve had the luxury to say, “Gee, I have time to go back and pretty this up a little bit.” And places where I’ve been able to write for a publication, you spend more time doing that than you would if it was just for your own professional growth.</p> <p>But I don’t think of that as art. I think <em>craft</em> is really the right word for it. You can make a chair, and it’s good looking, but it’s mostly functional – it’s a chair. [p319]</p> </blockquote> <h2 id="guy-steele">Guy Steele</h2> <p>On the difficulty of getting a program right:</p> <blockquote> <p><strong>Steele:</strong> I’ll give you another example – suppose I were to tell my smart computer, “OK, I’ve got this address book and I want the addresses to always be in sorted order,” and it responds by throwing away everything but the first entry. Now the address book is sorted. But that’s not what you wanted. It turns out that just specifying something as simple as “a list is in sorted order and I haven’t lost any of the data and nothing has been duplicated” is actually a fairly tricky specification to write. [p361]</p> </blockquote> <h2 id="dan-ingalls">Dan Ingalls</h2> <p>Was a nice read, though I could not find anything particularly interesting worth sharing. Nevertheless, along the lines Seibel says something that I have never heard of:</p> <blockquote> <p><strong>Seibel:</strong> Alan Kay has said that both Lisp and Smalltalk have the problem that they’re so good they eat their children. If you had known Lisp, then Smalltalk would have been the first eaten child. [p378]</p> </blockquote> <h2 id="l-peter-deutsch">L Peter Deutsch</h2> <p>On getting data structures right:</p> <blockquote> <p><strong>Deutsch:</strong> … if you get the data structures and their invariants right, most of the code will just kind of write itself. [p420]</p> </blockquote> <p>Conceptualization of software and memory pointers:</p> <blockquote> <p><strong>Deutsch:</strong> … I don’t look around and see anything that looks like an address or a pointer. We have objects; we don’t have these weird things that computer scientists misname “objects.”</p> <p><strong>Seibel:</strong> To say nothing of the scale. Two to the 64th of anything is a lot, and things happening billions of times a second is fast.</p> <p><strong>Deutsch:</strong> But that doesn’t bother us here in the real world. You know Avogadro’s number, right? Ten to the 23rd? So, we’re looking here around at a world that has incredible numbers of little things all clumped together and happening at the same time. It doesn’t bother us because the world is such that you don’t have to understand this table at a subatomic level. The physical properties of matter are such that 99.9 percent of the time you can understand it in aggregate. And everything you have to know about it, you can understand from dealing with it in aggregate. To a great extent, that is not true in the world of software.</p> <p>People keep trying to do modularization structures for software. And the state of that art has been improving over time, but it’s still, in my opinion, very far away from the ease with which we look around and see things that have, whatever it is, 10 to the 23rd atoms in them, and it doesn’t even faze us.</p> <p>Software is a discipline of detail, and that is a deep, horrendous fundamental problem with software. Until we understand how to conceptualize and organize software in a way that we don’t have to think about how every little piece interacts with every other piece, things are not going to get a whole lot better. And we’re very far from being there. [p424]</p> </blockquote> <h2 id="ken-thompson">Ken Thompson</h2> <p>On teaching:</p> <blockquote> <p><strong>Thompson:</strong> … I love the teaching: the hard work of a first class, the fun of the second class. Then the misery of the third. [p455]</p> </blockquote> <p>What I am supposed to do and what I am actually doing:</p> <blockquote> <p><strong>Thompson:</strong> We were supposed to be doing basic research but there was some basic research we should be doing and some basic research we shouldn’t be doing. And just coming out of the ashes of MULTICS, operating systems was one of those basic research things we shouldn’t be doing. Because we tried it, it didn’t work, it was a huge failure, it was expensive; let’s drop it. So I kind of expected that for what I was doing I was going to eventually get fired. I didn’t. [p458]</p> </blockquote> <p>Code rots:</p> <blockquote> <p><strong>Thompson:</strong> Code by itself almost rots and it’s gotta be rewritten. Even when nothing has changed, for some reason it rots. [p460]</p> </blockquote> <p>10 percent of the work:</p> <blockquote> <p><strong>Thompson:</strong> NELIAC was a system-programming version of Algol 58.</p> <p>Seibel: Was Bliss also from that era?</p> <p><strong>Thompson:</strong> Bliss I think was after. And their emphasis was trying to compile well. I think it was pretty clear from the beginning that you shouldn’t kill yourself compiling well. You should do well but not really good. And the reason is that in the time it takes you to go from well to really good, Moore’s law has already surpassed you. You can pick up 10 percent but while you’re picking up that 10 percent, computers have gotten twice as fast and maybe with some other stuff that matters more for optimization, like caches. I think it’s largely a waste of time to do really well. It’s really hard; you generate as many bugs as you fix. You should stop, not take that extra 100 percent of time to do 10 percent of the work. [p462]</p> </blockquote> <p>Writing an OS to test a file system:</p> <blockquote> <p><strong>Seibel:</strong> So you basically wrote an OS so you’d have a better environment to test your file system.</p> <p><strong>Thompson:</strong> Yes. Halfway through there that I realized it was a real time- sharing system. I was writing the shell to drive the file system. And then I was writing a couple other programs that drove the file system. And right about there I said, “All I need is an editor and I’ve got an operating system.” [p465]</p> </blockquote> <p>Economics of deciding on introducing a bag:</p> <blockquote> <p><strong>Thompson:</strong> Certainly every time I’ve written one of these non-compare subroutine calls, strcpy and stuff like that, I know that I’m writing a bug. And I somehow take the economic decision of whether the bug is worth the extra arguments. [p468]</p> </blockquote> <p>On testing:</p> <blockquote> <p><strong>Thompson:</strong> … Mostly just regression tests.</p> <p><strong>Seibel:</strong> By things that are harder to test, you mean things like device drivers or networking protocols?</p> <p><strong>Thompson:</strong> Well, they’re run all the time when you’re actually running an operating system.</p> <p><strong>Seibel:</strong> So you figure you’ll shake the bugs out that way?</p> <p><strong>Thompson:</strong> Oh, absolutely. I mean, what’s better as a test of an operating system than people beating on it? [p469]</p> </blockquote> <p>Code at Google:</p> <blockquote> <p><strong>Thompson:</strong> I guess way more than 50 percent of the code is the what-if kind. [p473]</p> </blockquote> <p>On literate programming:</p> <blockquote> <p><strong>Seibel:</strong> When I interviewed him, Knuth said the key to technical writing is to say everything twice in complementary ways. So I think he sees that as a feature of literate programming, not a bug.</p> <p><strong>Thompson:</strong> Well if you have two ways, one of them is real: what the machine executes. [p477]</p> </blockquote> <h2 id="fran-allen">Fran Allen</h2> <p>What makes a program beautiful?</p> <blockquote> <p><strong>Allen:</strong> That it is a simple straightforward solution to a problem; that has some intrinsic structure and obviousness about it that isn’t obvious from the problem itself. [p489]</p> </blockquote> <h2 id="bernie-cosell">Bernie Cosell</h2> <p>Should we teach Knuth to students?</p> <blockquote> <p><strong>Cosell:</strong> I would not teach students Knuth per se for two reasons. First, it’s got all this mathematical stuff where he’s not just trying to present the algorithms but to derive whether they’re good or bad. I’m not sure you need that. I understand a little bit of it and I’m not sure I need any of it. But getting a feel for what’s fast and what’s slow and when, that’s an important thing to do even if you don’t know how much faster or how much slower.</p> <p>The second problem is once students get sensitive to that, they get too clever by half. They start optimizing little parts of the program because, “This is the ideal place to do an AB unbalanced 2-3 double reverse backward pointer cube thing and I always wanted to write one of those.” So they spend a week or two tuning an obscure part of a program that doesn’t need anything, which is now more complicated and didn’t make the program any better. So they need a tempered understanding that there are all these algorithms, how they work, and how to apply them. It’s really more of a case of how to pick the right one for the job you’re trying to do as opposed to knowing that this one is an order n-cubed plus three and this one is just order n-squared times four. [p527]</p> </blockquote> <p>Writing programs and learning how to program:</p> <blockquote> <p><strong>Cosell:</strong> The binary bits are what computers want and the text file is for me. I would get people – bright, really good people, right out of college, tops of their classes – on one of my projects. And they would know all about programming and I would give them some piece of the project to work on. And we would start crossing swords at our project-review meetings. They would say, “Why are you complaining about the fact that I have my global variables here, that I’m not doing this, that you don’t like the way the subroutines are laid out? The program works.”</p> <p>They’d be stunned when I tell them, “I don’t care that the program works. The fact that you’re working here at all means that I expect you to be able to write programs that work. Writing programs that work is a skilled craft and you’re good at it. Now, you have to learn how to program.” [p543]</p> </blockquote> <p>Convictions:</p> <blockquote> <p><strong>Cosell:</strong> I had two convictions, which actually served me well: that programs ought to make sense and there are very, very few inherently hard problems. [p549]</p> </blockquote> <p>How long is it going to take you to put this change in?</p> <blockquote> <p><strong>Cosell:</strong> So when they ask, “How long is it going to take you to put this change in?” you have three answers. The first is the absolute shortest way, changing the one line of code. The second answer is how long it would be using my simple rule of rewriting the subroutine as if you were not going to make that mistake. Then the third answer is how long if you fix that bug if you were actually writing this subroutine in the better version of the program. [p550]</p> </blockquote> <p>Artistry in programming:</p> <blockquote> <p><strong>Cosell:</strong> Part of what I call the artistry of the computer program is how easy it is for future people to be able to change it without breaking it. [p555]</p> </blockquote> <p>Difficulty of programming and C:</p> <blockquote> <p><strong>Cosell:</strong> … programmers just can’t be careful enough. They don’t see all the places. And C makes too many places. Too scary for me, and I guess it’s fair to say I’ve programmed C only about five years less than Ken has. We’re not in the same league, but I have a long track record with C and know how difficult it is and I think C is a big part of the problem. [p559]</p> </blockquote> <p>75 million run-of-the-mill programmers and Java:</p> <blockquote> <p><strong>Cosell:</strong> When I first messed with Java – this was when it was little baby language, of course – I said, “Oh, this is just another one of those languages to help not-so-good programmers go down the straight and narrow by restricting what they can do.” But maybe we’ve come to a point where that’s the right thing. Maybe the world has gotten so dangerous you can’t have a good, flexible language that one percent or two percent of the programmers will use to make great art because the world is now populated with 75 million run-of-the-mill programmers building these incredibly complicated applications and they need more help than that. So maybe Java’s the right thing. I don’t know. [p560]</p> </blockquote> <p>Not-so-good programmers and C:</p> <blockquote> <p><strong>Cosell:</strong> I don’t want to say that C has outlived its usefulness, but I think it was used by too many good programmers so that now not-good-enough programmers are using it to build applications and the bottom line is they’re not good enough and they can’t. Maybe C is the perfect language for really good systems programmers, but unfortunately not-so-good systems and applications programmers are using it and they shouldn’t be. [p560]</p> </blockquote> <h2 id="donald-knuth">Donald Knuth</h2> <p>Teaching a class, writing a book, and programming:</p> <blockquote> <p><strong>Knuth:</strong> I could teach classes full-time and write a book full-time but software required so much attention to detail. It filled that much of my brain to the exclusion of other stuff. So it gave me a special admiration for people who do large software projects – I would never have guessed it without having been faced with that myself. [p572]</p> </blockquote> <p>Why isn’t everybody a super programmer and super writer?</p> <blockquote> <p><strong>Knuth:</strong> Now, why hasn’t this spread over the whole world and why isn’t everybody doing it? I’m not sure who it was who hit the nail on the head – I think it was Jon Bentley. Simplified it is like this: only two percent of the world’s population is born to be super programmers. And only two percent of the population is born to be super writers. And Knuth is expecting everybody to be both. [p574]</p> </blockquote> <p>Use of pointers in C:</p> <blockquote> <p><strong>Knuth:</strong> To me one of the most important revolutions in programming languages was the use of pointers in the C language. When you have nontrivial data structures, you often need one part of the structure to point to another part, and people played around with different ways to put that into a higher- level language. Tony Hoare, for example, had a pretty nice clean system but the thing that the C language added – which at first I thought was a big mistake and then it turned out I loved it – was that when x is a pointer and then you say, x + 1 , that doesn’t mean one more byte after x but it means one more node after x , depending on what x points to: if it points to a big node, x + 1 jumps by a large amount; if x points to a small thing, x + 1 just moves a little. That, to me, is one of the most amazing improvements in notation. [p585]</p> </blockquote> <p>I did not know about Knuth’s <em>change files</em>. But it seemed like an inconvenient overkill:</p> <blockquote> <p><strong>Knuth:</strong> I had written TeX and Metafont and people started asking for it. And they had 200 or 300 combinations of programming language and operating system and computer, so I wanted to make it easy to adapt my code to anybody’s system. So we came up with the solution that I would write a master program that worked at Stanford and then there was this add-on called a change file which could customize it to anybody else’s machine.</p> <p>A change file is a very simple thing. It consists of a bunch of little blobs of changes. Each change starts out with a few lines of code. You match until you find the first line in the master file that agrees with the first line of your change. When you get to the end of the part of the change that was supposed to match the master file, then comes the part which says, “Replace that by these lines instead.” [p586]</p> <p>The extreme example of this was when TeX was adapted to Unicode. They had a change file maybe 10 times as long as the master program. In other words, they changed from an 8-bit program to a 16-bit program but instead of going through and redoing my master program, they were so into change files that they just wrote their whole draft of what they called Omega as change files, as a million lines of change files to TeX’s 20,000 lines of code or something. So that’s the extreme. [p587]</p> </blockquote> <p>Is programming fun any more?</p> <blockquote> <p><strong>Knuth:</strong> So there’s that change and then there’s the change that I’m really worried about: that the way a lot of programming goes today isn’t any fun because it’s just plugging in magic incantations – combine somebody else’s software and start it up. It doesn’t have much creativity. I’m worried that it’s becoming too boring because you don’t have a chance to do anything much new. [p594]</p> </blockquote> <p>Code reading:</p> <blockquote> <p><strong>Knuth:</strong> … don’t only read the people who code like you. [p601]</p> </blockquote> tag:vy.github.io,2016-08-12://blog/post/2016/08/12/hotspot-heapdump-threadump/ Programmatically Taking Heap and Thread Dumps in HotSpot 2016-08-12T17:53:00Z 2016-08-12T17:53:00Z <p>While taking heap and thread dumps are one click away using modern JVM toolset, in many cases the deployment environment access restrictions render these options unusable. Hence, you might end up exposing these functionalities in certain ways like an internal REST interface. This implies a new nasty obstacle: You need to know how to programmatically take heap and thread dumps in a Java application. Unfortunately, there does not exist a standard interface to access these functionalities within the VM as of date. But if you are only concerned about HotSpot, then you are in luck!</p> <h1 id="heap-dumps">Heap Dumps</h1> <p>For heap dumps, once you get your teeth into a <a href="https://docs.oracle.com/javase/8/docs/jre/api/management/extension/com/sun/management/HotSpotDiagnosticMXBean.html">HotSpotDiagnosticMXBean</a>, you are safe to go. It already exposes a <a href="https://docs.oracle.com/javase/8/docs/jre/api/management/extension/com/sun/management/HotSpotDiagnosticMXBean.html#dumpHeap-java.lang.String-boolean-">dumpHeap()</a> method ready to be used.</p> <pre><code class="language-java"><span class="kn">import</span> <span class="nn">com.sun.management.HotSpotDiagnosticMXBean</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">javax.management.MBeanServer</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.io.File</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.io.IOException</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.lang.management.ManagementFactory</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">enum</span> <span class="n">HotSpotHeapDumps</span> <span class="o">{;</span> <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">HotSpotDiagnosticMXBean</span> <span class="n">HOT_SPOT_DIAGNOSTIC_MX_BEAN</span> <span class="o">=</span> <span class="n">getHotspotDiagnosticMxBean</span><span class="o">();</span> <span class="kd">private</span> <span class="kd">static</span> <span class="n">HotSpotDiagnosticMXBean</span> <span class="nf">getHotspotDiagnosticMxBean</span><span class="o">()</span> <span class="o">{</span> <span class="n">MBeanServer</span> <span class="n">server</span> <span class="o">=</span> <span class="n">ManagementFactory</span><span class="o">.</span><span class="na">getPlatformMBeanServer</span><span class="o">();</span> <span class="k">try</span> <span class="o">{</span> <span class="k">return</span> <span class="n">ManagementFactory</span><span class="o">.</span><span class="na">newPlatformMXBeanProxy</span><span class="o">(</span> <span class="n">server</span><span class="o">,</span> <span class="n">HOT_SPOT_DIAGNOSTIC_MX_BEAN_NAME</span><span class="o">,</span> <span class="n">HotSpotDiagnosticMXBean</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">IOException</span> <span class="n">error</span><span class="o">)</span> <span class="o">{</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="s">"failed getting Hotspot Diagnostic MX bean"</span><span class="o">,</span> <span class="n">error</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">create</span><span class="o">(</span><span class="n">File</span> <span class="n">file</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">live</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span> <span class="n">HOT_SPOT_DIAGNOSTIC_MX_BEAN</span><span class="o">.</span><span class="na">dumpHeap</span><span class="o">(</span><span class="n">file</span><span class="o">.</span><span class="na">getAbsolutePath</span><span class="o">(),</span> <span class="n">live</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>The second argument of <code>dumpHeap</code> denotes live objects, that is, objects that are reachable from others.</p> <p>Note that many real-world Java applications occupy quite some memory. As a result of this, created heap dump generally end up consuming significant amount of disk space. You need to come up with your own custom clean up mechanism to tackle this problem. (For instance, in a JAX-RS resource, you can purpose a custom <code>MessageBodyWriter</code> to delete the file after writing the entire file to the output stream.)</p> <h1 id="thread-dumps">Thread Dumps</h1> <p>When you think first about thread dumps, they just contain simple plain text data.</p> <pre><code>2016-08-12 18:40:46 Full thread dump OpenJDK 64-Bit Server VM (25.76-b198 mixed mode): "RMI TCP Connection(266)-127.0.0.1" #24884 daemon prio=9 os_prio=0 tid=0x00007f9474010000 nid=0x2cee runnable [0x00007f941571b000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked &lt;0x00000005c086e8b0&gt; (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(FilterInputStream.java:83) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$83/628845041.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - &lt;0x00000005c0489198&gt; (a java.util.concurrent.ThreadPoolExecutor$Worker) "JobScheduler FJ pool 0/4" #24883 daemon prio=6 os_prio=0 tid=0x00007f946415d800 nid=0x2ced waiting on condition [0x00007f94093d2000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for &lt;0x00000005d8a5f9e0&gt; (a jsr166e.ForkJoinPool) at jsr166e.ForkJoinPool.awaitWork(ForkJoinPool.java:1756) at jsr166e.ForkJoinPool.scan(ForkJoinPool.java:1694) at jsr166e.ForkJoinPool.runWorker(ForkJoinPool.java:1642) at jsr166e.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:108) Locked ownable synchronizers: - None </code></pre> <p>Unfortunately, thread dumps do not have a standard syntax. While there are various ways to produce this output, thread dump analysis software does not play well with them. For instance, <a href="https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c">IBM Thread and Monitor Dump Analyzer for Java</a> cannot parse thread dumps created by VisualVM using JMX. At the end of the day, I always needed to fall back to a HotSpot thread dump.</p> <p><code>tools.jar</code> shipped with JDKs (&gt;=1.6) provide the magical <a href="http://www.docjar.com/docs/api/sun/tools/attach/HotSpotVirtualMachine.html">HotSpotVirtualMachine</a> class containing our saviour <code>remoteDataDump()</code> method. First add the following lines to your <code>pom.xml</code>:</p> <pre><code class="language-xml"><span class="nt">&lt;dependencyManagement&gt;</span> <span class="nt">&lt;dependencies&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.sun<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>tools<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${java.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;scope&gt;</span>system<span class="nt">&lt;/scope&gt;</span> <span class="nt">&lt;systemPath&gt;</span>${tools.jar}<span class="nt">&lt;/systemPath&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="nt">&lt;/dependencies&gt;</span> <span class="nt">&lt;/dependencyManagement&gt;</span> <span class="nt">&lt;profiles&gt;</span> <span class="c">&lt;!-- tools.jar path for GNU/Linux and Windows --&gt;</span> <span class="nt">&lt;profile&gt;</span> <span class="nt">&lt;id&gt;</span>default-tools.jar<span class="nt">&lt;/id&gt;</span> <span class="nt">&lt;activation&gt;</span> <span class="nt">&lt;file&gt;</span> <span class="nt">&lt;exists&gt;</span>${java.home}/../lib/tools.jar<span class="nt">&lt;/exists&gt;</span> <span class="nt">&lt;/file&gt;</span> <span class="nt">&lt;/activation&gt;</span> <span class="nt">&lt;properties&gt;</span> <span class="nt">&lt;tools.jar&gt;</span>${java.home}/../lib/tools.jar<span class="nt">&lt;/tools.jar&gt;</span> <span class="nt">&lt;/properties&gt;</span> <span class="nt">&lt;/profile&gt;</span> <span class="c">&lt;!-- tools.jar path for OSX --&gt;</span> <span class="nt">&lt;profile&gt;</span> <span class="nt">&lt;id&gt;</span>default-tools.jar-mac<span class="nt">&lt;/id&gt;</span> <span class="nt">&lt;activation&gt;</span> <span class="nt">&lt;file&gt;</span> <span class="nt">&lt;exists&gt;</span>${java.home}/../Classes/classes.jar<span class="nt">&lt;/exists&gt;</span> <span class="nt">&lt;/file&gt;</span> <span class="nt">&lt;/activation&gt;</span> <span class="nt">&lt;properties&gt;</span> <span class="nt">&lt;tools.jar&gt;</span>${java.home}/../Classes/classes.jar<span class="nt">&lt;/tools.jar&gt;</span> <span class="nt">&lt;/properties&gt;</span> <span class="nt">&lt;/profile&gt;</span> <span class="nt">&lt;/profiles&gt;</span></code></pre> <p>Then the rest is a matter of accessing to <code>HotSpotVirtualMachine</code> class:</p> <pre><code class="language-java"><span class="kn">import</span> <span class="nn">com.google.common.io.ByteStreams</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.sun.management.HotSpotDiagnosticMXBean</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.sun.tools.attach.AttachNotSupportedException</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">com.sun.tools.attach.VirtualMachine</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">sun.tools.attach.HotSpotVirtualMachine</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.io.IOException</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.io.InputStream</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.lang.management.ManagementFactory</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">enum</span> <span class="n">HotSpotThreadDumps</span> <span class="o">{;</span> <span class="kd">public</span> <span class="n">String</span> <span class="nf">create</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">AttachNotSupportedException</span><span class="o">,</span> <span class="n">IOException</span> <span class="o">{</span> <span class="c1">// Get the PID of the current JVM process.</span> <span class="n">String</span> <span class="n">selfName</span> <span class="o">=</span> <span class="n">ManagementFactory</span><span class="o">.</span><span class="na">getRuntimeMXBean</span><span class="o">().</span><span class="na">getName</span><span class="o">();</span> <span class="n">String</span> <span class="n">selfPid</span> <span class="o">=</span> <span class="n">selfName</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">selfName</span><span class="o">.</span><span class="na">indexOf</span><span class="o">(</span><span class="sc">'@'</span><span class="o">));</span> <span class="c1">// Attach to the VM.</span> <span class="n">VirtualMachine</span> <span class="n">vm</span> <span class="o">=</span> <span class="n">VirtualMachine</span><span class="o">.</span><span class="na">attach</span><span class="o">(</span><span class="n">selfPid</span><span class="o">);</span> <span class="n">HotSpotVirtualMachine</span> <span class="n">hotSpotVm</span> <span class="o">=</span> <span class="o">(</span><span class="n">HotSpotVirtualMachine</span><span class="o">)</span> <span class="n">vm</span><span class="o">;</span> <span class="c1">// Request a thread dump.</span> <span class="k">try</span> <span class="o">(</span><span class="n">InputStream</span> <span class="n">inputStream</span> <span class="o">=</span> <span class="n">hotSpotVm</span><span class="o">.</span><span class="na">remoteDataDump</span><span class="o">())</span> <span class="o">{</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">bytes</span> <span class="o">=</span> <span class="n">ByteStreams</span><span class="o">.</span><span class="na">toByteArray</span><span class="o">(</span><span class="n">inputStream</span><span class="o">);</span> <span class="k">return</span> <span class="k">new</span> <span class="nf">String</span><span class="o">(</span><span class="n">bytes</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>You finished writing this code, you clicked on the Run button of the IDE, and it worked like a charm. This get you so excited that you wanted to add this functionality to your JEE service! Or better: Turn this into a JAR and pass it to your client’s machine and watch them take their part in the joy of thread-dump-oriented debugging! And this is what you get in return:</p> <pre><code>java.lang.NoClassDefFoundError: com/sun/tools/attach/AttachNotSupportedException </code></pre> <p>Which indicates that you did not pay attention my words: <em><code>tools.jar</code> is shipped with JDKs.</em> So neither your flashy JEE application server, nor your client’s machine has a JDK, but a JRE. Rings a bell? Yes, you indeed can add <code>tools.jar</code> into the final WAR/JAR of your project:</p> <pre><code class="language-xml"><span class="nt">&lt;build&gt;</span> <span class="nt">&lt;plugins&gt;</span> <span class="c">&lt;!-- copy tools.jar from JAVA_HOME --&gt;</span> <span class="nt">&lt;plugin&gt;</span> <span class="nt">&lt;groupId&gt;</span>org.apache.maven.plugins<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>maven-dependency-plugin<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;executions&gt;</span> <span class="nt">&lt;execution&gt;</span> <span class="nt">&lt;id&gt;</span>copy-system-dependencies<span class="nt">&lt;/id&gt;</span> <span class="nt">&lt;phase&gt;</span>prepare-package<span class="nt">&lt;/phase&gt;</span> <span class="nt">&lt;goals&gt;</span> <span class="nt">&lt;goal&gt;</span>copy-dependencies<span class="nt">&lt;/goal&gt;</span> <span class="nt">&lt;/goals&gt;</span> <span class="nt">&lt;configuration&gt;</span> <span class="nt">&lt;outputDirectory&gt;</span>${project.build.directory}/${project.build.finalName}/WEB-INF/lib<span class="nt">&lt;/outputDirectory&gt;</span> <span class="nt">&lt;includeScope&gt;</span>system<span class="nt">&lt;/includeScope&gt;</span> <span class="nt">&lt;/configuration&gt;</span> <span class="nt">&lt;/execution&gt;</span> <span class="nt">&lt;/executions&gt;</span> <span class="nt">&lt;/plugin&gt;</span> <span class="nt">&lt;/plugins&gt;</span> <span class="nt">&lt;/build&gt;</span></code></pre> <p>Note that this approach incorporates a JDK-specific JAR into your application and assumes that the application will run on a HotSpot VM. But unfortunately this is the only way that I know of to produce a thread dump that works with thread dump analysis software. If you don’t have such a need and just want a crude JMX generated thread dump, check out <a href="https://java.net/projects/visualvm/sources/svn/content/branches/release134/visualvm/jmx/src/com/sun/tools/visualvm/jmx/impl/JmxSupport.java">JmxSupport.java</a> shipped with VisualVM.</p> tag:vy.github.io,2016-07-20://blog/post/2016/07/20/rxjava-backpressure/ Callback Blocking for Back-Pressure in RxJava 2016-07-20T20:32:00Z 2016-07-20T20:32:00Z <p>In a reactive application, you don’t necessarily have control over the production and/or consumption rate of certain streams. This speed mismatch can cause severe and hard to find bugs, which might be overlooked in development environments while bringing in the entire system down in production.</p> <h1 id="life-without-back-pressure">Life Without Back-Pressure</h1> <p>Consider the following example:</p> <pre><code class="language-java"><span class="kn">import</span> <span class="nn">com.google.common.base.Throwables</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">rx.Observable</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.concurrent.atomic.AtomicInteger</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">enum</span> <span class="n">NoBackPressure</span> <span class="o">{;</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="kt">long</span> <span class="n">producePeriod</span> <span class="o">=</span> <span class="mi">100</span><span class="o">;</span> <span class="kt">long</span> <span class="n">consumePeriod</span> <span class="o">=</span> <span class="mi">300</span><span class="o">;</span> <span class="n">AtomicInteger</span> <span class="n">pendingTaskCount</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">AtomicInteger</span><span class="o">();</span> <span class="c1">// Create a fast producer emitting an infinite number of items.</span> <span class="n">createStream</span><span class="o">(</span><span class="n">producePeriod</span><span class="o">,</span> <span class="kc">true</span><span class="o">,</span> <span class="nl">pendingTaskCount:</span><span class="o">:</span><span class="n">incrementAndGet</span><span class="o">)</span> <span class="o">.</span><span class="na">flatMap</span><span class="o">(</span><span class="n">ignored</span> <span class="o">-&gt;</span> <span class="c1">// Create a slow consumer emitting just one item.</span> <span class="n">createStream</span><span class="o">(</span><span class="n">consumePeriod</span><span class="o">,</span> <span class="kc">false</span><span class="o">,</span> <span class="nl">pendingTaskCount:</span><span class="o">:</span><span class="n">decrementAndGet</span><span class="o">))</span> <span class="o">.</span><span class="na">take</span><span class="o">(</span><span class="mi">5</span><span class="o">)</span> <span class="o">.</span><span class="na">toBlocking</span><span class="o">()</span> <span class="o">.</span><span class="na">last</span><span class="o">();</span> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">format</span><span class="o">(</span><span class="s">"pending task count: %d\n"</span><span class="o">,</span> <span class="n">pendingTaskCount</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="o">}</span> <span class="kd">private</span> <span class="kd">static</span> <span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">Observable</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">createStream</span><span class="o">(</span><span class="kt">long</span> <span class="n">pausePeriodMillis</span><span class="o">,</span> <span class="kt">boolean</span> <span class="n">infinite</span><span class="o">,</span> <span class="n">Supplier</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">body</span><span class="o">)</span> <span class="o">{</span> <span class="k">return</span> <span class="n">Observable</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">subscriber</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="k">new</span> <span class="nf">Thread</span><span class="o">()</span> <span class="o">{</span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">run</span><span class="o">()</span> <span class="o">{</span> <span class="k">do</span> <span class="o">{</span> <span class="n">pause</span><span class="o">(</span><span class="n">pausePeriodMillis</span><span class="o">);</span> <span class="n">T</span> <span class="n">next</span> <span class="o">=</span> <span class="n">body</span><span class="o">.</span><span class="na">get</span><span class="o">();</span> <span class="n">subscriber</span><span class="o">.</span><span class="na">onNext</span><span class="o">(</span><span class="n">next</span><span class="o">);</span> <span class="o">}</span> <span class="k">while</span> <span class="o">(</span><span class="n">infinite</span> <span class="o">&amp;&amp;</span> <span class="o">!</span><span class="n">subscriber</span><span class="o">.</span><span class="na">isUnsubscribed</span><span class="o">());</span> <span class="o">}</span> <span class="o">}.</span><span class="na">start</span><span class="o">();</span> <span class="o">});</span> <span class="o">}</span> <span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">pause</span><span class="o">(</span><span class="kt">long</span> <span class="n">millis</span><span class="o">)</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="n">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="n">millis</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">InterruptedException</span> <span class="n">error</span><span class="o">)</span> <span class="o">{</span> <span class="n">Throwables</span><span class="o">.</span><span class="na">propagate</span><span class="o">(</span><span class="n">error</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>What’s going on really here? The fast producer is an observable emitting an item every 100ms and then incrementing the <code>pendingTaskCount</code>. Subsequently, the emitted item is <code>flatMap</code>ed into another consumer observable emitting an item every 300ms and then decrementing the <code>pendingTaskCount</code>. That is, yet another simple producer-consumer pipeline. Finally, we ask for the first 5 items emitted out of the pipeline. Can you guess the program output? Or let me rephrase the question: Do you expect <code>pendingTaskCount</code> to be non-zero? Unfortunately, yes. It is 3 in this case. Let’s shed some more light into it:</p> <p><img src="prod-cons-pipeline.jpg" alt="Producer-Consumer Pipeline"></p> <p>As my spectular drawing skills depict above, during the completion of the final 5th item, the producer generates 3 other items which later on get processed by the slow consumer. So you have 3 extra threads lingering in the background hogging both memory and processing resources. (Why 3? Because <code>consumePeriod / producePeriod = 3</code>.) While 3 seems like an innocent and hence negligible magnitude, this speed unalignment can get a lot more worse once you deploy the application to production. (Yes, it did in our case at work.) What do I exactly mean by worse? <em>If we would set <code>consumePeriod</code> to 10s, and <code>producePeriod</code> to 10ms, then there will be 1000 threads running in the background at any particular point in time!</em></p> <h1 id="rx-has-a-word-to-say">Rx Has a Word To Say!</h1> <p>In a nutshell, we need to come up with a way to regulate the production pace in line with the consumption. We can either do this by an on-demand producer (<em>reactive pull</em>) or blocking the producer itself (<em>callstack blocking</em>). (Both in its <a href="https://github.com/ReactiveX/RxJava/wiki/Backpressure">official wiki</a> and <a href="http://stackoverflow.com/documentation/rx-java/2341/backpressure">Stack Overflow Documentation</a>, RxJava has quite some juice on the subject.)</p> <h2 id="discarding-the-over-production">Discarding the Over-Production</h2> <p>Three common methods provided out of the box by RxJava for dealing with back-pressure are <code>onBackpressureBuffer</code>, <code>onBackpressureDrop</code>, and <code>onBackpressureLatest</code>. While they definitely do the trick, rather than regulating the production speed, they just discard emitted items by the producer under certain back-pressure circumstances. (I am keeping experimental RxJava &gt;1.0 feature <code>onBackpressureBlock</code> out of this discussion due to its ambiguous future and known track record of holding a potential to introduce dead-locks.)</p> <h2 id="reactive-pull">Reactive Pull</h2> <p>RxJava has one more bullet in the hand though: <a href="http://stackoverflow.com/documentation/rxjava/2341/backpressure">SyncOnSubscribe</a>. This almost orphan, totally undocumented prodigy, provides the necessary harness to create <em>stateful</em> and <em>on-demand</em> producers:</p> <pre><code class="language-java"><span class="n">SyncOnSubscribe</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">InputStream</span><span class="o">&gt;</span> <span class="n">binaryReader</span> <span class="o">=</span> <span class="n">SyncOnSubscribe</span><span class="o">.</span><span class="na">createStateful</span><span class="o">(</span> <span class="c1">// Create the initial state. (Invoked per subscriber.)</span> <span class="o">()</span> <span class="o">-&gt;</span> <span class="k">new</span> <span class="nf">FileInputStream</span><span class="o">(</span><span class="s">"data.bin"</span><span class="o">),</span> <span class="c1">// Upon request, emit a new item and return the new state.</span> <span class="o">(</span><span class="n">inputStream</span><span class="o">,</span> <span class="n">output</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="kt">int</span> <span class="kt">byte</span> <span class="o">=</span> <span class="n">inputStream</span><span class="o">.</span><span class="na">read</span><span class="o">();</span> <span class="k">if</span> <span class="o">(</span><span class="kt">byte</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="o">)</span> <span class="n">output</span><span class="o">.</span><span class="na">onCompleted</span><span class="o">()</span> <span class="k">else</span> <span class="n">output</span><span class="o">.</span><span class="na">onNext</span><span class="o">(</span><span class="kt">byte</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">IOException</span> <span class="n">ex</span><span class="o">)</span> <span class="o">{</span> <span class="n">output</span><span class="o">.</span><span class="na">onError</span><span class="o">(</span><span class="n">ex</span><span class="o">);</span> <span class="o">}</span> <span class="k">return</span> <span class="n">inputStream</span><span class="o">;</span> <span class="o">},</span> <span class="c1">// Perform final clean-up using the state. (Invoked upon unsubscription.)</span> <span class="n">inputStream</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="n">inputStream</span><span class="o">.</span><span class="na">close</span><span class="o">();</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">IOException</span> <span class="n">error</span><span class="o">)</span> <span class="o">{</span> <span class="n">RxJavaHooks</span><span class="o">.</span><span class="na">onError</span><span class="o">(</span><span class="n">error</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="o">);</span> <span class="n">Observable</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">observableBinaryReader</span> <span class="o">=</span> <span class="n">Observable</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">binaryReader</span><span class="o">);</span></code></pre> <p>Awesome! We are done, right? Unfortunately not. In RxJava, unless you specify otherwise, <a href="http://reactivex.io/RxJava/javadoc/rx/Subscriber.html#request(long)">every consumer tries to pull <code>Long.MAX_VALUE</code> items from the observable it is subscribed to</a>. You can change this beaviour by overriding this value:</p> <pre><code class="language-java"><span class="n">observableBinaryReader</span><span class="o">.</span><span class="na">subscribe</span><span class="o">(</span><span class="k">new</span> <span class="n">Subscriber</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;()</span> <span class="o">{</span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onStart</span><span class="o">()</span> <span class="o">{</span> <span class="n">request</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span> <span class="c1">// Request 1 item on start up.</span> <span class="o">}</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onNext</span><span class="o">(</span><span class="n">Integer</span> <span class="n">v</span><span class="o">)</span> <span class="o">{</span> <span class="n">compute</span><span class="o">(</span><span class="n">v</span><span class="o">);</span> <span class="n">request</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span> <span class="c1">// Request a new item after consuming one.</span> <span class="o">}</span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onError</span><span class="o">(</span><span class="n">Throwable</span> <span class="n">error</span><span class="o">)</span> <span class="o">{</span> <span class="n">error</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span> <span class="o">}</span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onCompleted</span><span class="o">()</span> <span class="o">{</span> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"Done!"</span><span class="o">);</span> <span class="o">}</span> <span class="o">});</span></code></pre> <p>In other words, the subscriber needs to be aware of the producer-consumer pace mismatch and align them explicitly by limiting the number of requested items. To the best of my knowledge, it is not possible to enforce the subscriber to specify the number of requested items. You just need to hope that the next programmer consuming your <code>Observable&lt;T&gt;</code> will be able to figure out the back-pressure problem and override the <code>request(Long.MAX_VALUE)</code> behaviour. (But you know that he won’t, right?)</p> <p>As a matter of fact, <em>reactive pull</em> does not provide a solution for our over-productive observable example, which just blindly emits items by ignoring the consumer pace. We need a way to block the production according to the consumption rate. And Rx literature has already got a term for this approach: <em>Callstack Blocking</em>.</p> <h2 id="callstack-blocking">Callstack Blocking</h2> <p>Shamelessly copying from the <a href="https://github.com/ReactiveX/RxJava/wiki/Backpressure#callstack-blocking-as-a-flow-control-alternative-to-backpressure">RxJava wiki</a>:</p> <blockquote> <p>Another way of handling an over-productive <code>Observable</code> is to block the callstack (parking the thread that governs the over-productive <code>Observable</code>). This has the disadvantage of going against the <em>reactive</em> and non-blocking model of Rx. However this can be a viable option if the problematic <code>Observable</code> is on a thread that can be blocked safely. Currently RxJava does not expose any operators to facilitate this.</p> </blockquote> <p>But the good news is, you can implement this yourself. Let me walk-through you how to do it.</p> <h1 id="stack-your-own-back-pressure">Stack Your Own Back-Pressure</h1> <p>Let me introduce you to the poor man’s back-pressure queue.</p> <pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="kt">long</span> <span class="n">producePeriod</span> <span class="o">=</span> <span class="mi">100</span><span class="o">;</span> <span class="kt">long</span> <span class="n">consumePeriod</span> <span class="o">=</span> <span class="mi">300</span><span class="o">;</span> <span class="n">AtomicInteger</span> <span class="n">pendingTaskCount</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">AtomicInteger</span><span class="o">();</span> <span class="c1">// The token queue for producer-consumer pipeline.</span> <span class="n">BlockingQueue</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">tokens</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayBlockingQueue</span><span class="o">&lt;&gt;(</span> <span class="mi">1</span><span class="o">,</span> <span class="c1">// Number of tokens allowed.</span> <span class="kc">false</span><span class="o">,</span> <span class="c1">// fair? (preserve the FIFO order?)</span> <span class="n">Collections</span><span class="o">.</span><span class="na">singleton</span><span class="o">(</span><span class="mi">1</span><span class="o">));</span> <span class="c1">// Initial tokens.</span> <span class="n">createStream</span><span class="o">(</span><span class="n">producePeriod</span><span class="o">,</span> <span class="kc">true</span><span class="o">,</span> <span class="o">()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="n">pendingTaskCount</span><span class="o">.</span><span class="na">incrementAndGet</span><span class="o">();</span> <span class="c1">// Try to acquire a token from the queue.</span> <span class="k">try</span> <span class="o">{</span> <span class="k">return</span> <span class="n">tokens</span><span class="o">.</span><span class="na">take</span><span class="o">();</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">InterruptedException</span> <span class="n">error</span><span class="o">)</span> <span class="o">{</span> <span class="k">throw</span> <span class="n">Throwables</span><span class="o">.</span><span class="na">propagate</span><span class="o">(</span><span class="n">error</span><span class="o">);</span> <span class="o">}</span> <span class="o">})</span> <span class="o">.</span><span class="na">flatMap</span><span class="o">(</span><span class="n">token</span> <span class="o">-&gt;</span> <span class="n">createStream</span><span class="o">(</span><span class="n">consumePeriod</span><span class="o">,</span> <span class="kc">false</span><span class="o">,</span> <span class="o">()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="n">pendingTaskCount</span><span class="o">.</span><span class="na">decrementAndGet</span><span class="o">();</span> <span class="c1">// Push the token back into the queue.</span> <span class="k">try</span> <span class="o">{</span> <span class="n">tokens</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">token</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">InterruptedException</span> <span class="n">error</span><span class="o">)</span> <span class="o">{</span> <span class="k">throw</span> <span class="n">Throwables</span><span class="o">.</span><span class="na">propagate</span><span class="o">(</span><span class="n">error</span><span class="o">);</span> <span class="o">}</span> <span class="k">return</span> <span class="kc">null</span><span class="o">;</span> <span class="o">}))</span> <span class="o">.</span><span class="na">take</span><span class="o">(</span><span class="mi">5</span><span class="o">)</span> <span class="o">.</span><span class="na">toBlocking</span><span class="o">()</span> <span class="o">.</span><span class="na">last</span><span class="o">();</span> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">format</span><span class="o">(</span><span class="s">"pending task count: %d\n"</span><span class="o">,</span> <span class="n">pendingTaskCount</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="o">}</span></code></pre> <p>Here we use a blocking queue to implement a token storage where producers acquire from and consumers release to. This way we create a way to communicate the back-pressure from consumers to the producer. Initially there is just a single token. Producer acquires this token and emits an item. Note that the upcoming producer call of the thread will block since there are no tokens left in the queue. Next, consumer emits an item and releases the token back into the queue. Now the blocked thread can proceed and emit a new item and so on. By limiting the number of tokens initially available within the queue, we put an upper limit on the number of concurrent consumptions. This version of our producer-consumer pipeline reports that <code>pendingTaskCount</code> is 1, which is independent of the producer/consumer speed mismatch.</p> <h1 id="back-pressure-for-the-masses">Back-Pressure for the Masses</h1> <p>Can we avoid having a global reference to the token storage and make it explicit in the return type of the observable signature? Consider the following two interfaces:</p> <pre><code class="language-java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">BackPressuredFactory</span> <span class="o">{</span> <span class="nd">@Nonnull</span> <span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">BackPressured</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">acquire</span><span class="o">(</span><span class="nd">@Nullable</span> <span class="n">T</span> <span class="n">instance</span><span class="o">);</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">interface</span> <span class="nc">BackPressured</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="o">{</span> <span class="nd">@Nullable</span> <span class="n">T</span> <span class="nf">getValue</span><span class="o">();</span> <span class="kt">void</span> <span class="nf">release</span><span class="o">();</span> <span class="o">}</span></code></pre> <p>A factory for creating instances of <code>BackPressured&lt;T&gt;</code>, which encapsulates a value associated with a certain token that is supposed to be released. Let’s try to put them into use:</p> <pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="kt">long</span> <span class="n">producePeriod</span> <span class="o">=</span> <span class="mi">100</span><span class="o">;</span> <span class="kt">long</span> <span class="n">consumePeriod</span> <span class="o">=</span> <span class="mi">300</span><span class="o">;</span> <span class="n">AtomicInteger</span> <span class="n">pendingTaskCount</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">AtomicInteger</span><span class="o">();</span> <span class="n">BackPressuredFactory</span> <span class="n">backPressuredFactory</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">BackPressuredFactoryImpl</span><span class="o">(</span> <span class="mi">1</span><span class="o">,</span> <span class="c1">// Number of concurrent tokens allowed.</span> <span class="mi">5000</span><span class="o">);</span> <span class="c1">// Max. acquire/release timeout in milliseconds.</span> <span class="n">createStream</span><span class="o">(</span><span class="n">producePeriod</span><span class="o">,</span> <span class="kc">true</span><span class="o">,</span> <span class="o">()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="n">pendingTaskCount</span><span class="o">.</span><span class="na">incrementAndGet</span><span class="o">();</span> <span class="c1">// Wrap the next item with a BackPressured&lt;T&gt; instance.</span> <span class="n">BackPressured</span><span class="o">&lt;</span><span class="n">Void</span><span class="o">&gt;</span> <span class="n">next</span> <span class="o">=</span> <span class="n">backPressuredFactory</span><span class="o">.</span><span class="na">acquire</span><span class="o">(</span><span class="kc">null</span><span class="o">);</span> <span class="k">return</span> <span class="n">next</span><span class="o">;</span> <span class="o">})</span> <span class="o">.</span><span class="na">flatMap</span><span class="o">(</span><span class="n">backPressuredToken</span> <span class="o">-&gt;</span> <span class="n">createStream</span><span class="o">(</span><span class="n">consumePeriod</span><span class="o">,</span> <span class="kc">false</span><span class="o">,</span> <span class="o">()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="n">pendingTaskCount</span><span class="o">.</span><span class="na">decrementAndGet</span><span class="o">();</span> <span class="c1">// Getting the value out of the back-pressured token.</span> <span class="k">return</span> <span class="n">backPressuredToken</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span> <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span> <span class="c1">// Release the token.</span> <span class="n">backPressuredToken</span><span class="o">.</span><span class="na">release</span><span class="o">();</span> <span class="o">}</span> <span class="o">}))</span> <span class="o">.</span><span class="na">take</span><span class="o">(</span><span class="mi">5</span><span class="o">)</span> <span class="o">.</span><span class="na">toBlocking</span><span class="o">()</span> <span class="o">.</span><span class="na">last</span><span class="o">();</span> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">format</span><span class="o">(</span><span class="s">"pending task count: %d\n"</span><span class="o">,</span> <span class="n">pendingTaskCount</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="o">}</span></code></pre> <p>In a nutshell, we encapsulate every item of type <code>T</code> that producer emits into a <code>BackPressured&lt;T&gt;</code> instance. <code>BackPressuredFactory</code> contains the token storage. Given these requirements a sample implementation of these interfaces can be given as follows:</p> <pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.slf4j.Logger</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">org.slf4j.LoggerFactory</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.List</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.concurrent.ArrayBlockingQueue</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.concurrent.BlockingQueue</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.concurrent.TimeUnit</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.stream.Collectors</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.stream.IntStream</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">static</span> <span class="n">com</span><span class="o">.</span><span class="na">google</span><span class="o">.</span><span class="na">common</span><span class="o">.</span><span class="na">base</span><span class="o">.</span><span class="na">Preconditions</span><span class="o">.</span><span class="na">checkArgument</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">BackPressuredFactoryImpl</span> <span class="kd">implements</span> <span class="n">BackPressuredFactory</span> <span class="o">{</span> <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">Logger</span> <span class="n">LOGGER</span> <span class="o">=</span> <span class="n">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="n">BackPressuredFactoryImpl</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">BlockingQueue</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">tokens</span><span class="o">;</span> <span class="kd">private</span> <span class="kd">final</span> <span class="kt">long</span> <span class="n">timeoutMillis</span><span class="o">;</span> <span class="kd">public</span> <span class="nf">BackPressuredFactoryImpl</span><span class="o">(</span><span class="kt">int</span> <span class="n">bufferSize</span><span class="o">,</span> <span class="kt">long</span> <span class="n">timeoutMillis</span><span class="o">)</span> <span class="o">{</span> <span class="n">checkArgument</span><span class="o">(</span><span class="n">bufferSize</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="o">,</span> <span class="s">"bufferSize &gt; 0, found: %d"</span><span class="o">,</span> <span class="n">bufferSize</span><span class="o">);</span> <span class="n">checkArgument</span><span class="o">(</span><span class="n">timeoutMillis</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="o">,</span> <span class="s">"timeoutMillis &gt; 0, found: %d"</span><span class="o">,</span> <span class="n">timeoutMillis</span><span class="o">);</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">initialTokens</span> <span class="o">=</span> <span class="n">IntStream</span><span class="o">.</span><span class="na">range</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="n">bufferSize</span><span class="o">).</span><span class="na">boxed</span><span class="o">().</span><span class="na">collect</span><span class="o">(</span><span class="n">Collectors</span><span class="o">.</span><span class="na">toList</span><span class="o">());</span> <span class="k">this</span><span class="o">.</span><span class="na">tokens</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayBlockingQueue</span><span class="o">&lt;&gt;(</span><span class="n">bufferSize</span><span class="o">,</span> <span class="kc">false</span><span class="o">,</span> <span class="n">initialTokens</span><span class="o">);</span> <span class="k">this</span><span class="o">.</span><span class="na">timeoutMillis</span> <span class="o">=</span> <span class="n">timeoutMillis</span><span class="o">;</span> <span class="n">LOGGER</span><span class="o">.</span><span class="na">trace</span><span class="o">(</span><span class="s">"initialized (bufferSize={}, timeoutMillis={})"</span><span class="o">,</span> <span class="n">bufferSize</span><span class="o">,</span> <span class="n">timeoutMillis</span><span class="o">);</span> <span class="o">}</span> <span class="nd">@Nonnull</span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">BackPressured</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">acquire</span><span class="o">(</span><span class="nd">@Nullable</span> <span class="n">T</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span> <span class="n">LOGGER</span><span class="o">.</span><span class="na">trace</span><span class="o">(</span><span class="s">"acquiring (peekedToken={})"</span><span class="o">,</span> <span class="n">tokens</span><span class="o">.</span><span class="na">peek</span><span class="o">());</span> <span class="k">try</span> <span class="o">{</span> <span class="n">Integer</span> <span class="n">token</span> <span class="o">=</span> <span class="n">tokens</span><span class="o">.</span><span class="na">poll</span><span class="o">(</span><span class="n">timeoutMillis</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">MILLISECONDS</span><span class="o">);</span> <span class="k">if</span> <span class="o">(</span><span class="n">token</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="s">"token acquisition timeout"</span><span class="o">);</span> <span class="k">return</span> <span class="k">new</span> <span class="n">BackPressuredImpl</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;(</span><span class="n">tokens</span><span class="o">,</span> <span class="n">timeoutMillis</span><span class="o">,</span> <span class="n">token</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">InterruptedException</span> <span class="n">error</span><span class="o">)</span> <span class="o">{</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="s">"token acquisition failure"</span><span class="o">,</span> <span class="n">error</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>And here is <code>BackPressured&lt;T&gt;</code>:</p> <pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.slf4j.Logger</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">org.slf4j.LoggerFactory</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.concurrent.BlockingQueue</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.concurrent.TimeUnit</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">static</span> <span class="n">com</span><span class="o">.</span><span class="na">google</span><span class="o">.</span><span class="na">common</span><span class="o">.</span><span class="na">base</span><span class="o">.</span><span class="na">Preconditions</span><span class="o">.</span><span class="na">checkArgument</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">static</span> <span class="n">com</span><span class="o">.</span><span class="na">google</span><span class="o">.</span><span class="na">common</span><span class="o">.</span><span class="na">base</span><span class="o">.</span><span class="na">Preconditions</span><span class="o">.</span><span class="na">checkNotNull</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">BackPressuredImpl</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="kd">implements</span> <span class="n">BackPressured</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="o">{</span> <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">Logger</span> <span class="n">LOGGER</span> <span class="o">=</span> <span class="n">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="n">BackPressuredImpl</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">BlockingQueue</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">tokens</span><span class="o">;</span> <span class="kd">private</span> <span class="kd">final</span> <span class="kt">long</span> <span class="n">timeoutMillis</span><span class="o">;</span> <span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">token</span><span class="o">;</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">T</span> <span class="n">value</span><span class="o">;</span> <span class="kd">public</span> <span class="nf">BackPressuredImpl</span><span class="o">(</span><span class="nd">@Nonnull</span> <span class="n">BlockingQueue</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">tokens</span><span class="o">,</span> <span class="kt">long</span> <span class="n">timeoutMillis</span><span class="o">,</span> <span class="kt">int</span> <span class="n">token</span><span class="o">,</span> <span class="nd">@Nullable</span> <span class="n">T</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span> <span class="n">checkArgument</span><span class="o">(</span><span class="n">timeoutMillis</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="o">,</span> <span class="s">"timeoutMillis &gt; 0, found: %d"</span><span class="o">,</span> <span class="n">timeoutMillis</span><span class="o">);</span> <span class="k">this</span><span class="o">.</span><span class="na">tokens</span> <span class="o">=</span> <span class="n">checkNotNull</span><span class="o">(</span><span class="n">tokens</span><span class="o">,</span> <span class="s">"null tokens"</span><span class="o">);</span> <span class="k">this</span><span class="o">.</span><span class="na">timeoutMillis</span> <span class="o">=</span> <span class="n">timeoutMillis</span><span class="o">;</span> <span class="k">this</span><span class="o">.</span><span class="na">token</span> <span class="o">=</span> <span class="n">token</span><span class="o">;</span> <span class="k">this</span><span class="o">.</span><span class="na">value</span> <span class="o">=</span> <span class="n">value</span><span class="o">;</span> <span class="n">LOGGER</span><span class="o">.</span><span class="na">trace</span><span class="o">(</span><span class="s">"initialized (token={})"</span><span class="o">,</span> <span class="n">token</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span> <span class="o">}</span> <span class="nd">@Nullable</span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="n">T</span> <span class="nf">getValue</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">value</span><span class="o">;</span> <span class="o">}</span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">release</span><span class="o">()</span> <span class="o">{</span> <span class="n">LOGGER</span><span class="o">.</span><span class="na">trace</span><span class="o">(</span><span class="s">"releasing (token={})"</span><span class="o">,</span> <span class="n">token</span><span class="o">);</span> <span class="k">try</span> <span class="o">{</span> <span class="k">if</span> <span class="o">(!</span><span class="n">tokens</span><span class="o">.</span><span class="na">offer</span><span class="o">(</span><span class="n">token</span><span class="o">,</span> <span class="n">timeoutMillis</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">MILLISECONDS</span><span class="o">))</span> <span class="o">{</span> <span class="n">String</span> <span class="n">message</span> <span class="o">=</span> <span class="n">String</span><span class="o">.</span><span class="na">format</span><span class="o">(</span><span class="s">"token release timeout (timeoutMillis=%d, token=%d)"</span><span class="o">,</span> <span class="n">timeoutMillis</span><span class="o">,</span> <span class="n">token</span><span class="o">);</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="n">message</span><span class="o">);</span> <span class="o">}</span> <span class="n">LOGGER</span><span class="o">.</span><span class="na">trace</span><span class="o">(</span><span class="s">"released (token={})"</span><span class="o">,</span> <span class="n">token</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">InterruptedException</span> <span class="n">error</span><span class="o">)</span> <span class="o">{</span> <span class="n">String</span> <span class="n">message</span> <span class="o">=</span> <span class="n">String</span><span class="o">.</span><span class="na">format</span><span class="o">(</span><span class="s">"token release failure (timeoutMillis=%d, token=%d)"</span><span class="o">,</span> <span class="n">timeoutMillis</span><span class="o">,</span> <span class="n">token</span><span class="o">);</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="n">message</span><span class="o">,</span> <span class="n">error</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="o">}</span></code></pre> <h1 id="conclusion">Conclusion</h1> <p>Back-pressure is a significant aspect in every producer-consumer pipeline. It can be easily overlooked and holds a potential to break the system depending on the speed mismatch of the involved actors. In this post, I examined the problem in a sample RxJava application and provided a solution leveraging <em>callback blocking</em> approach that can be employed in almost any domain where the back-pressure needs to communicated. I hope you find it useful as well.</p> tag:vy.github.io,2016-03-02://blog/post/2016/03/02/optimize-dc/ Optimizing a Data Center Using Integer Programming 2016-03-02T19:32:00Z 2016-03-02T19:32:00Z <p><a href="https://hashcode.withgoogle.com/2015/tasks/hashcode2015_qualification_task.pdf">Optimize a Data Center</a> is a <strong>challenging</strong> programming problem presented in the Qualification Round of the <a href="https://hashcode.withgoogle.com/past_editions.html">Hash Code 2015</a>. Among 230 teams, <em>What’s in a name?</em> from <a href="http://www.ens.fr/">École normale supérieure</a> ranked first in the qualification, but third in the final round. By challenging, I mean it is not possible to come up with a deterministic polynomial-time optimal answer. I am not in a position to either provide a rigorous proof of its complexity or its reduction to a known NP-hard problem. But in this blog post I will investigate the following question: Can we provide an optimal solution using <a href="https://en.wikipedia.org/wiki/Integer_programming">integer programming</a>? In practice, that would allow us to come up with an optimal solution to small-sized problems. Without further ado, let’s start with the problem definition.</p> <h1 id="the-problem">The Problem</h1> <p>Here, I will present a brief summary of <a href="https://hashcode.withgoogle.com/2015/tasks/hashcode2015_qualification_task.pdf">the actual problem</a>. A data center is modeled as <strong>rows​</strong> of <strong>slots</strong> ​in which servers can be placed. And some of the slots are known to be <strong>unavailable</strong>.</p> <p><img src="rows.jpg" alt="Data center rows"></p> <p>Each <strong>server</strong> is characterized by its <strong>size</strong> and <strong>capacity​</strong>. Size is the number of consecutive slots occupied by the machine. Capacity is the total amount of CPU resources of the machine (an integer value).</p> <p><img src="servers.jpg" alt="Data center servers"></p> <p>Servers in a data center are also logically divided into <strong>pools</strong>. ​Each server belongs to exactly one pool. The capacity of a pool is the sum of the capacities of the available ​servers in that pool.</p> <p>The <strong>guaranteed capacity</strong> ​of a pool is the minimum capacity it will have when at most one data center row goes down. Given a schema of a data center and a list of available servers, the goal is to assign servers to slots within the rows​ and to logical pools ​so that the lowest guaranteed capacity​ of all pools is maximized.</p> <p>Consider the following data center schema and a list of available servers. For simplicity, it is assumed that server capacities are equal to server sizes.</p> <p><img src="example-prob.jpg" alt="Example problem"></p> <p>Following layout is a solution to the above given problem. Here different pools are denoted in distinct colors.</p> <p><img src="example-soln.jpg" alt="Example solution"></p> <h1 id="preliminaries">Preliminaries</h1> <p>Before modeling the IP (Integer Program), I will start with stating the problem input.</p> <ul> <li> <script type="math/tex">R</script> ​denotes the number of rows in the data center</li> <li> <script type="math/tex">S</script> denotes the number of slots in each row of the data center</li> <li> <script type="math/tex">U \leq RS</script> denotes the number of unavailable slots</li> <li> <script type="math/tex">P</script> denotes the number of pools to be created</li> <li> <script type="math/tex">M \leq RS</script> denotes the number of servers to be allocated</li> <li> <script type="math/tex">z_k</script> and <script type="math/tex">c_k</script> denote <script type="math/tex">k</script>th server’s respectively size and capacity</li> <li> <script type="math/tex">r_i</script> and <script type="math/tex">s_i</script> denote <script type="math/tex">i</script>th unavailable slot’s respectively row and slot indices</li> </ul> <p>While modeling the formulation, I will need to provide constraints to avoid placing servers to unavailable slots. Rather than doing that, <a href="http://kaygun.tumblr.com/">Atabey Kaygun</a> hinted me to represent the data in <strong>blocks</strong>. That is, instead of <script type="math/tex">U</script>, <script type="math/tex">r_i</script>, and <script type="math/tex">s_i</script>, I will transform this data into a single lookup table called <script type="math/tex">z(i, j)</script> that denotes the size of the available slots at <script type="math/tex">i</script>th row and <script type="math/tex">j</script>th block. For instance, consider the following layout:</p> <p><img src="blocks.jpg" alt="Example blocks"></p> <p>Here blocks will look as follows:</p> <script type="math/tex; mode=display">% <![CDATA[ \begin{align} [z(0, 0)] &= [10] \\ [z(1, 0), z(1, 1)] &= [5,3] \\ [z(2, 0), z(2, 1), z(2, 2)] &= [3, 4, 1] \\ [z(3, 0)] &= [6] \end{align} %]]></script> <h1 id="the-integer-programming-model">The Integer Programming Model</h1> <p>For simplicity, I will adopt the following index notation:</p> <ul> <li> <script type="math/tex">i</script> denotes row indices (<script type="math/tex">0 \leq i \leq R</script>)</li> <li> <script type="math/tex">j</script> denotes block (not slot!) indices (varies per row)</li> <li> <script type="math/tex">k</script> denotes server indices (<script type="math/tex">0 \leq k \leq M</script>)</li> <li> <script type="math/tex">\ell</script> denotes pool indices (<script type="math/tex">0 \leq \ell \leq P</script>)</li> </ul> <p>Given that <script type="math/tex">z(i, j)</script> denotes the available blocks, the IP model can be defined as follows:</p> <script type="math/tex; mode=display">% <![CDATA[ \begin{align} \text{maximize} & \quad \min_\ell g(\ell) \quad \text{(minimum of pool guaranteed capacities)} \\ \text{subject to} & \quad \sum_\ell a(i, j, k, \ell) \leq 1, \, \forall k \quad \text{(a server can be assigned to at most 1 pool and block)} \\ & \quad \sum_{k,\ell} z_k \, a(i, j, k, \ell) \leq z(i, j), \, \forall i, j \quad \text{(total size of servers within a block cannot exceed block's size)} \\ \text{where} & \quad a(i, j, k, \ell) = \begin{cases} 1 & \quad \text{if } k \text{th server is assigned to } (i, j) \text{th block and } \ell \text{th pool} \\ 0 & \quad \text{otherwise} \\ \end{cases} \\ & \quad g(\ell) = \min_i g(\ell, i) \quad \text{(guaranteed capacity of } \ell \text{th pool)} \\ & \quad g(\ell, i) = \sum_{i', j, k} c_k \, a(i', j, k, \ell) - \sum_{j, k} c_k \, a(i, j, k, \ell) \quad \text{(guaranteed capacity of } \ell \text{th pool for } i \text{th row)} \\ \end{align} %]]></script> <h1 id="avoiding-minimax-constraint">Avoiding Minimax Constraint</h1> <p>The presented IP model contains a minimax objective, which to the best of my knowledge is not tractable by popular linear programming optimizers, such as <a href="http://www-01.ibm.com/software/commerce/optimization/cplex-optimizer/">CPLEX</a> or <a href="http://lpsolve.sourceforge.net/">lpsolve</a>. But I have a trick in my pocket to tackle that. Let’s assume that we know the optimal objective, say <script type="math/tex">g^*</script>. Then we can model the entire IP as follows:</p> <script type="math/tex; mode=display">% <![CDATA[ \begin{align} \text{maximize} & \quad 1 \quad \text{(a dummy objective)} \\ & \quad \sum_\ell a(i, j, k, \ell) \leq 1, \, \forall k \\ & \quad \sum_{k,\ell} z_k \, a(i, j, k, \ell) \leq z(i, j), \, \forall i, j \\ & \quad g(\ell, i) \geq g^*, \, \forall \ell, i \quad \text{(guaranteed capacity must be greater than or equal to } g^* \text{)} \\ \end{align} %]]></script> <p>What this model states is this: I am not interested in the optimization objective, return me the first found feasible solution. That is, the optimizer will return us the first <script type="math/tex">a(i, j, k, \ell)</script> variable set the moment it finds a feasible solution satisfying <script type="math/tex">g(\ell, i) \geq g^*, \, \forall \ell, i</script> constraints.</p> <p>Now things are getting interesting. If we can find bounds to <script type="math/tex">g^*</script>, than we can use these bounds to bisect the optimal <script type="math/tex">g^*</script>! For the lower bound, we know that <script type="math/tex">g_i = 0 \leq g^*</script>. The upper bound is a little bit tricky, but we can come up with a quite loose bound: <script type="math/tex">g_f = \frac{1}{P} \sum_k c_k \gg g^*</script>. (I will not go into details of how to come up with a stricter upper bound.) So by picking <script type="math/tex">g^* \in (g_i, g_f)</script> we can bisect <strong>the optimal guaranteed capacity</strong>.</p> <h1 id="the-solver">The Solver</h1> <p>For testing purposes, I wrote a simple <a href="https://gist.github.com/vy/9689cb122a84be22d454">Python script</a> that reads an input problem file and calls CPLEX iteratively. A sample output of the script is as follows:</p> <pre><code>$ ./dc.py ./cplex.sh prob/dc-min.in soln/dc-min 0 2016-03-03 08:35:25 DEBUG reading setup: prob/dc-min.in 2016-03-03 08:35:25 DEBUG R=2, S=5, U=1, P=2, M=5 2016-03-03 08:35:25 INFO solving for 0 &lt;= g=8 &lt; 16 2016-03-03 08:35:25 DEBUG writing problem: soln/dc-min-8.lp 2016-03-03 08:35:25 DEBUG running solver 2016-03-03 08:35:25 DEBUG writing cplex output: soln/dc-min-8.out 2016-03-03 08:35:25 DEBUG no solution 2016-03-03 08:35:25 DEBUG stepping back 2016-03-03 08:35:25 INFO solving for 0 &lt;= g=4 &lt; 8 2016-03-03 08:35:25 DEBUG writing problem: soln/dc-min-4.lp 2016-03-03 08:35:25 DEBUG running solver 2016-03-03 08:35:25 DEBUG writing cplex output: soln/dc-min-4.out 2016-03-03 08:35:25 DEBUG solution score: 5 2016-03-03 08:35:25 DEBUG writing solution: soln/dc-min-4.soln 2016-03-03 08:35:25 DEBUG stepping forward 2016-03-03 08:35:25 INFO solving for 5 &lt;= g=6 &lt; 8 2016-03-03 08:35:25 DEBUG writing problem: soln/dc-min-6.lp 2016-03-03 08:35:25 DEBUG running solver 2016-03-03 08:35:25 DEBUG writing cplex output: soln/dc-min-6.out 2016-03-03 08:35:25 DEBUG no solution 2016-03-03 08:35:25 DEBUG stepping back 2016-03-03 08:35:25 INFO solving for 5 &lt;= g=5 &lt; 6 2016-03-03 08:35:25 DEBUG writing problem: soln/dc-min-5.lp 2016-03-03 08:35:25 DEBUG running solver 2016-03-03 08:35:25 DEBUG writing cplex output: soln/dc-min-5.out 2016-03-03 08:35:25 DEBUG solution score: 5 2016-03-03 08:35:25 DEBUG writing solution: soln/dc-min-5.soln 2016-03-03 08:35:25 DEBUG stepping forward </code></pre> <p>It also outputs intermediate IP formulation files, CPLEX output, and solution file. The format of the problem and solution files are detailed in <a href="https://hashcode.withgoogle.com/2015/tasks/hashcode2015_qualification_task.pdf">the official problem description</a>.</p> tag:vy.github.io,2015-11-27://blog/post/2015/11/27/maven-protobuf/ Compiling Protocol Buffers Sources in Maven 2015-11-27T08:49:00Z 2015-11-27T08:49:00Z <blockquote> <p><strong>TL;DR</strong> – This post explains how to compile Protocol Buffers schemas into Java sources in Maven. Over the course of time, there appeared plugins like <a href="https://github.com/os72/protoc-jar-maven-plugin">protoc-jar-maven-plugin</a>. Nevertheless, the steps below still present a value to understand the necessary plumbing and some best practices (e.g., shading) that are not covered by the plugins.</p> </blockquote> <p>Java build systems have always been a second class citizen for <a href="https://developers.google.com/protocol-buffers">Protocol Buffers</a>. As is the case for many other Java serialization frameworks (e.g., <a href="https://capnproto.org/">Cap’n Proto</a>, <a href="http://google.github.io/flatbuffers/">FlatBuffers</a>), Protocol Buffers does not provide a native Java compiler which you can inject into Maven dependencies and invoke a plugin to compile <code>.proto</code> files into <code>.java</code> sources. Hence, programmers needed to have <code>protoc</code> (Proto Buffers Compiler) binary on their development machine and call this platform-dependent binary during Maven build. This totally violates the environment independent build of a project. Fortunately, Google releases platform-specific <code>protoc</code> binaries in the form of Maven artifacts.</p> <p><img src="protoc-artifacts.png" alt="Maven Artifacts for Proto Buffers Compiler Binary"></p> <p>We can add these artifacts as a compile-time dependency to our Maven project and invoke the platform-dependent binary to compile <code>.proto</code> sources.</p> <h1 id="preliminaries">Preliminaries</h1> <p>Let’s start with defining certain properties for library versions and input/output directories for the Protocol Buffers compiler.</p> <pre><code class="language-xml"><span class="nt">&lt;properties&gt;</span> <span class="c">&lt;!-- protobuf paths --&gt;</span> <span class="nt">&lt;protobuf.input.directory&gt;</span>${project.basedir}/src/main/proto<span class="nt">&lt;/protobuf.input.directory&gt;</span> <span class="nt">&lt;protobuf.output.directory&gt;</span>${project.build.directory}/generated-sources<span class="nt">&lt;/protobuf.output.directory&gt;</span> <span class="c">&lt;!-- library versions --&gt;</span> <span class="nt">&lt;build-helper-maven-plugin.version&gt;</span>1.9.1<span class="nt">&lt;/build-helper-maven-plugin.version&gt;</span> <span class="nt">&lt;maven-antrun-plugin.version&gt;</span>1.8<span class="nt">&lt;/maven-antrun-plugin.version&gt;</span> <span class="nt">&lt;maven-dependency-plugin.version&gt;</span>2.10<span class="nt">&lt;/maven-dependency-plugin.version&gt;</span> <span class="nt">&lt;maven-shade-plugin.version&gt;</span>2.4.2<span class="nt">&lt;/maven-shade-plugin.version&gt;</span> <span class="nt">&lt;os-maven-plugin.version&gt;</span>1.4.1.Final<span class="nt">&lt;/os-maven-plugin.version&gt;</span> <span class="nt">&lt;protobuf.version&gt;</span>3.0.0-beta-1<span class="nt">&lt;/protobuf.version&gt;</span> <span class="nt">&lt;/properties&gt;</span></code></pre> <h1 id="protocol-buffers-java-api">Protocol Buffers Java API</h1> <p><code>protoc</code> compiles <code>.proto</code> files into <code>.java</code> files such that the generated sources rely on certain common classes. These classes are provided by <code>protobuf-java</code> artifact:</p> <pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.google.protobuf<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>protobuf-java<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${protobuf.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span></code></pre> <h1 id="detecting-operating-system">Detecting Operating System</h1> <p><code>protoc</code> Maven artifact is provided in various platform-specific <em>classifications</em>: <code>linux-x86_32</code>, <code>linux-x86_64</code>, <code>osx-x86_32</code>, <code>osx-x86_64</code>, <code>windows-x86_32</code>, <code>windows-x86_64</code>. In order to pick the right artifact, we will employ <code>os.detected.classifier</code> property exposed by <code>os-maven-plugin</code>:</p> <pre><code class="language-xml"><span class="nt">&lt;build&gt;</span> <span class="nt">&lt;extensions&gt;</span> <span class="c">&lt;!-- provides os.detected.classifier (i.e. linux-x86_64, osx-x86_64) property --&gt;</span> <span class="nt">&lt;extension&gt;</span> <span class="nt">&lt;groupId&gt;</span>kr.motd.maven<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>os-maven-plugin<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${os-maven-plugin.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/extension&gt;</span> <span class="nt">&lt;/extensions&gt;</span> <span class="c">&lt;!-- ... --&gt;</span> <span class="nt">&lt;/build&gt;</span></code></pre> <h1 id="downloading-platform-specific-protocol-buffers-compiler">Downloading Platform-Specific Protocol Buffers Compiler</h1> <p>We will use <code>maven-dependency-plugin</code> to download the platform-specific <code>protoc</code> binary suitable for the current build platform and copy it into <code>project.build.directory</code>.</p> <pre><code class="language-xml"><span class="c">&lt;!-- copy protoc binary into build directory --&gt;</span> <span class="nt">&lt;plugin&gt;</span> <span class="nt">&lt;groupId&gt;</span>org.apache.maven.plugins<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>maven-dependency-plugin<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${maven-dependency-plugin.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;executions&gt;</span> <span class="nt">&lt;execution&gt;</span> <span class="nt">&lt;id&gt;</span>copy-protoc<span class="nt">&lt;/id&gt;</span> <span class="nt">&lt;phase&gt;</span>generate-sources<span class="nt">&lt;/phase&gt;</span> <span class="nt">&lt;goals&gt;</span> <span class="nt">&lt;goal&gt;</span>copy<span class="nt">&lt;/goal&gt;</span> <span class="nt">&lt;/goals&gt;</span> <span class="nt">&lt;configuration&gt;</span> <span class="nt">&lt;artifactItems&gt;</span> <span class="nt">&lt;artifactItem&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.google.protobuf<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>protoc<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${protobuf.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;classifier&gt;</span>${os.detected.classifier}<span class="nt">&lt;/classifier&gt;</span> <span class="nt">&lt;type&gt;</span>exe<span class="nt">&lt;/type&gt;</span> <span class="nt">&lt;overWrite&gt;</span>true<span class="nt">&lt;/overWrite&gt;</span> <span class="nt">&lt;outputDirectory&gt;</span>${project.build.directory}<span class="nt">&lt;/outputDirectory&gt;</span> <span class="nt">&lt;/artifactItem&gt;</span> <span class="nt">&lt;/artifactItems&gt;</span> <span class="nt">&lt;/configuration&gt;</span> <span class="nt">&lt;/execution&gt;</span> <span class="nt">&lt;/executions&gt;</span> <span class="nt">&lt;/plugin&gt;</span></code></pre> <p>Note how we employed <code>os.detected.classifier</code> variable provided by <code>os-maven-plugin</code> to inject the platform-specific binary dependency.</p> <h1 id="generating-protocol-buffers-java-sources">Generating Protocol Buffers Java Sources</h1> <p>Now we have our <code>protoc</code> binary in <code>project.build.directory</code>. We can use <code>maven-antrun-plugin</code> plugin to execute <code>protoc</code> for compiling <code>.proto</code> files into <code>.java</code> sources.</p> <pre><code class="language-xml"><span class="c">&lt;!-- compile proto buffer files using copied protoc binary --&gt;</span> <span class="nt">&lt;plugin&gt;</span> <span class="nt">&lt;groupId&gt;</span>org.apache.maven.plugins<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>maven-antrun-plugin<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${maven-antrun-plugin.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;executions&gt;</span> <span class="nt">&lt;execution&gt;</span> <span class="nt">&lt;id&gt;</span>exec-protoc<span class="nt">&lt;/id&gt;</span> <span class="nt">&lt;phase&gt;</span>generate-sources<span class="nt">&lt;/phase&gt;</span> <span class="nt">&lt;configuration&gt;</span> <span class="nt">&lt;target&gt;</span> <span class="nt">&lt;property</span> <span class="na">name=</span><span class="s">"protoc.filename"</span> <span class="na">value=</span><span class="s">"protoc-${protobuf.version}-${os.detected.classifier}.exe"</span><span class="nt">/&gt;</span> <span class="nt">&lt;property</span> <span class="na">name=</span><span class="s">"protoc.filepath"</span> <span class="na">value=</span><span class="s">"${project.build.directory}/${protoc.filename}"</span><span class="nt">/&gt;</span> <span class="nt">&lt;chmod</span> <span class="na">file=</span><span class="s">"${protoc.filepath}"</span> <span class="na">perm=</span><span class="s">"ugo+rx"</span><span class="nt">/&gt;</span> <span class="nt">&lt;mkdir</span> <span class="na">dir=</span><span class="s">"${protobuf.output.directory}"</span> <span class="nt">/&gt;</span> <span class="nt">&lt;path</span> <span class="na">id=</span><span class="s">"protobuf.input.filepaths.path"</span><span class="nt">&gt;</span> <span class="nt">&lt;fileset</span> <span class="na">dir=</span><span class="s">"${protobuf.input.directory}"</span><span class="nt">&gt;</span> <span class="nt">&lt;include</span> <span class="na">name=</span><span class="s">"**/*.proto"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/fileset&gt;</span> <span class="nt">&lt;/path&gt;</span> <span class="nt">&lt;pathconvert</span> <span class="na">pathsep=</span><span class="s">" "</span> <span class="na">property=</span><span class="s">"protobuf.input.filepaths"</span> <span class="na">refid=</span><span class="s">"protobuf.input.filepaths.path"</span><span class="nt">/&gt;</span> <span class="nt">&lt;exec</span> <span class="na">executable=</span><span class="s">"${protoc.filepath}"</span> <span class="na">failonerror=</span><span class="s">"true"</span><span class="nt">&gt;</span> <span class="nt">&lt;arg</span> <span class="na">value=</span><span class="s">"-I"</span><span class="nt">/&gt;</span> <span class="nt">&lt;arg</span> <span class="na">value=</span><span class="s">"${protobuf.input.directory}"</span><span class="nt">/&gt;</span> <span class="nt">&lt;arg</span> <span class="na">value=</span><span class="s">"--java_out"</span><span class="nt">/&gt;</span> <span class="nt">&lt;arg</span> <span class="na">value=</span><span class="s">"${protobuf.output.directory}"</span><span class="nt">/&gt;</span> <span class="nt">&lt;arg</span> <span class="na">line=</span><span class="s">"${protobuf.input.filepaths}"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/exec&gt;</span> <span class="nt">&lt;/target&gt;</span> <span class="nt">&lt;/configuration&gt;</span> <span class="nt">&lt;goals&gt;</span> <span class="nt">&lt;goal&gt;</span>run<span class="nt">&lt;/goal&gt;</span> <span class="nt">&lt;/goals&gt;</span> <span class="nt">&lt;/execution&gt;</span> <span class="nt">&lt;/executions&gt;</span> <span class="nt">&lt;/plugin&gt;</span></code></pre> <h1 id="adding-generated-sources-into-the-package">Adding Generated Sources into the Package</h1> <p><code>protoc</code> compiler placed the generated Java sources into <code>protobuf.output.directory</code>. We need to add the sources in this directory to the package:</p> <pre><code class="language-xml"><span class="c">&lt;!-- add generated proto buffer classes into the package --&gt;</span> <span class="nt">&lt;plugin&gt;</span> <span class="nt">&lt;groupId&gt;</span>org.codehaus.mojo<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>build-helper-maven-plugin<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${build-helper-maven-plugin.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;executions&gt;</span> <span class="nt">&lt;execution&gt;</span> <span class="nt">&lt;id&gt;</span>add-classes<span class="nt">&lt;/id&gt;</span> <span class="nt">&lt;phase&gt;</span>generate-sources<span class="nt">&lt;/phase&gt;</span> <span class="nt">&lt;goals&gt;</span> <span class="nt">&lt;goal&gt;</span>add-source<span class="nt">&lt;/goal&gt;</span> <span class="nt">&lt;/goals&gt;</span> <span class="nt">&lt;configuration&gt;</span> <span class="nt">&lt;sources&gt;</span> <span class="nt">&lt;source&gt;</span>${protobuf.output.directory}<span class="nt">&lt;/source&gt;</span> <span class="nt">&lt;/sources&gt;</span> <span class="nt">&lt;/configuration&gt;</span> <span class="nt">&lt;/execution&gt;</span> <span class="nt">&lt;/executions&gt;</span> <span class="nt">&lt;/plugin&gt;</span></code></pre> <h1 id="shading-protocol-buffers-package">Shading Protocol Buffers Package</h1> <p>Say you are done with your project, which includes <code>protobuf-java</code> version 3.0.0-beta-1 as a dependency. What if there is another package that is included as a direct or transitive Maven dependency and injects <code>protobuf-java</code> version 2.5.0? Then you are doomed; you will get a package version conflict. In order to avoid this problem, you can leverage <code>maven-shade-plugin</code> to <em>relocate</em> <code>com.google.protobuf</code> package contents to a private package within your project:</p> <pre><code class="language-xml"><span class="c">&lt;!-- shade protobuf to avoid version conflicts --&gt;</span> <span class="nt">&lt;plugin&gt;</span> <span class="nt">&lt;groupId&gt;</span>org.apache.maven.plugins<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>maven-shade-plugin<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${maven-shade-plugin.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;executions&gt;</span> <span class="nt">&lt;execution&gt;</span> <span class="nt">&lt;phase&gt;</span>package<span class="nt">&lt;/phase&gt;</span> <span class="nt">&lt;goals&gt;</span> <span class="nt">&lt;goal&gt;</span>shade<span class="nt">&lt;/goal&gt;</span> <span class="nt">&lt;/goals&gt;</span> <span class="nt">&lt;configuration&gt;</span> <span class="nt">&lt;relocations&gt;</span> <span class="nt">&lt;relocation&gt;</span> <span class="nt">&lt;pattern&gt;</span>com.google.protobuf<span class="nt">&lt;/pattern&gt;</span> <span class="nt">&lt;shadedPattern&gt;</span>${project.groupId}.${project.artifactId}.shaded.protobuf<span class="nt">&lt;/shadedPattern&gt;</span> <span class="nt">&lt;/relocation&gt;</span> <span class="nt">&lt;/relocations&gt;</span> <span class="nt">&lt;/configuration&gt;</span> <span class="nt">&lt;/execution&gt;</span> <span class="nt">&lt;/executions&gt;</span> <span class="nt">&lt;/plugin&gt;</span></code></pre> <p>This will relocate the contents of <code>com.google.protobuf</code> package to <code>${project.groupId}.${project.artifactId}.shaded.protobuf</code> and make the classes accessible under this namespace. That is, instead of using</p> <pre><code class="language-java"><span class="kn">import</span> <span class="nn">com.google.protobuf.*</span><span class="o">;</span></code></pre> <p>in your project, you should use the new relocated package name:</p> <pre><code class="language-java"><span class="kn">import</span> <span class="nn">groupId.artifactId.shaded.protobuf.*</span><span class="o">;</span></code></pre> <p>(Note that you need to replace <code>groupId</code> and <code>artifactId</code> literals in the Java code.)</p> <h1 id="conclusion">Conclusion</h1> <p>In this post, I tried to summarize the necessary set of steps to compile Protocol Buffers schema into Java classes using plain Maven magic. This is important to get a platform-independent build for a project.</p> <p>The absence of a proper Maven plugin to handle all these steps for us causes quite a bit of <code>pom.xml</code> pollution. Nevertheless, (de)serializers for the messaging medium are generally distributed in a separate artifact, hence this will probably end up being the entire content of your <code>pom.xml</code>.</p> tag:vy.github.io,2015-11-20://blog/post/2015/11/20/scala-method-with-java-reserved-keyword/ Calling a Scala Method with a Java Reserved Keyword 2015-11-20T07:00:00Z 2015-11-20T07:00:00Z <p>At work, there are a couple of JAX-RS microservices I develop and maintain. Fortunately, all of these services and their consumers are written in Scala. Up until now, I have never needed to access the service APIs from Java. But some other team recently needed to do that! And that was where shit hit the fan. Consider the following <code>ScalaApi</code> class:</p> <pre><code class="language-scala"><span class="k">class</span> <span class="nc">ScalaApi</span> <span class="o">{</span> <span class="k">def</span> <span class="n">default</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="s">"test"</span> <span class="o">}</span></code></pre> <p>Did you notice something wrong with the class methods? While it is perfectly legitimate to use <code>default</code> as a keyword in the JVM byte code, it is not allowed by the <a href="http://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html#jls-3.9">Java Language Specification 3.9</a>. (Another slightly related discussion: <a href="http://stackoverflow.com/questions/423994/reserved-words-as-variable-or-method-names">Reserved words as variable or method names</a>.) Hence, you cannot access it via a simple <code>new ScalaApi().default()</code> call in Java. To the best of my knowledge, your only safe bet becomes using Java’s Reflection API:</p> <pre><code class="language-java"><span class="kn">import</span> <span class="nn">java.lang.reflect.Method</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">Consumer</span> <span class="o">{</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="n">ScalaApi</span> <span class="n">scalaApi</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">ScalaApi</span><span class="o">();</span> <span class="n">String</span> <span class="n">scalaApiDefault</span> <span class="o">=</span> <span class="n">scalaApiDefault</span><span class="o">(</span><span class="n">scalaApi</span><span class="o">);</span> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"ScalaApi#default: "</span> <span class="o">+</span> <span class="n">scalaApiDefault</span><span class="o">);</span> <span class="o">}</span> <span class="kd">private</span> <span class="kd">static</span> <span class="n">String</span> <span class="nf">scalaApiDefault</span><span class="o">(</span><span class="n">ScalaApi</span> <span class="n">scalaApi</span><span class="o">)</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="n">Method</span> <span class="n">method</span> <span class="o">=</span> <span class="n">scalaApi</span><span class="o">.</span><span class="na">getClass</span><span class="o">().</span><span class="na">getDeclaredMethod</span><span class="o">(</span><span class="s">"default"</span><span class="o">);</span> <span class="k">return</span> <span class="o">(</span><span class="n">String</span><span class="o">)</span> <span class="n">method</span><span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="n">scalaApi</span><span class="o">);</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">NoSuchMethodException</span> <span class="o">|</span> <span class="n">InvocationTargetException</span> <span class="o">|</span> <span class="n">IllegalAccessException</span> <span class="n">cause</span><span class="o">)</span> <span class="o">{</span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">RuntimeException</span><span class="o">(</span><span class="n">cause</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>This issue raised the following question: <em>Where did I make a mistake?</em> I believe, it is not possible to come up with a universally legitimate API targeting all available languages that compile to JVM byte code. I think that is one of the reasons why programming languages provide constructs to use reserved keywords in the source code, such as <code>reserved</code> in Scala or <code>@reserved</code> in C#, etc.</p> tag:vy.github.io,2015-09-09://blog/post/2015/09/09/enforce-locking/ Enforcing a Locking Context on Variables in Scala 2015-09-09T18:56:00Z 2015-09-09T18:56:00Z <p>Accessing variables that need synchronization necessitates book keeping by programmers. Since it is not something explicitly enforced by the language mechanics, programmer needs to make sure that such variables are not accessed out of a synchronization scope. Consider the following innocent user store:</p> <pre><code class="language-scala"><span class="k">case</span> <span class="k">class</span> <span class="nc">User</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">manager</span><span class="k">:</span> <span class="kt">Boolean</span><span class="o">)</span> <span class="c1">// Use with caution! Can be accessed by multiple threads.</span> <span class="k">private</span> <span class="k">var</span> <span class="n">users</span><span class="k">:</span> <span class="kt">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">User</span><span class="o">]()</span></code></pre> <p>The safest approach would be encapsulating every access to <code>users</code> variables in a <code>synchronized</code> block:</p> <pre><code class="language-scala"><span class="k">def</span> <span class="n">save</span><span class="o">(</span><span class="n">user</span><span class="k">:</span> <span class="kt">User</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">synchronized</span> <span class="o">{</span> <span class="n">users</span> <span class="o">+=</span> <span class="n">user</span><span class="o">.</span><span class="n">id</span> <span class="o">-&gt;</span> <span class="n">user</span> <span class="o">}</span> <span class="k">def</span> <span class="n">findById</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="n">synchronized</span> <span class="o">{</span> <span class="n">users</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">id</span><span class="o">)</span> <span class="o">}</span></code></pre> <p>You can even go further and use a <code>ReadWriteLock</code> to boost read-only queries:</p> <pre><code class="language-scala"><span class="k">private</span> <span class="k">val</span> <span class="n">lock</span><span class="k">:</span> <span class="kt">ReadWriteLock</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ReentrantReadWriteLock</span><span class="o">()</span> <span class="k">def</span> <span class="n">save</span><span class="o">(</span><span class="n">user</span><span class="k">:</span> <span class="kt">User</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span> <span class="n">lock</span><span class="o">.</span><span class="n">writeLock</span><span class="o">().</span><span class="n">lock</span><span class="o">()</span> <span class="k">try</span> <span class="o">{</span> <span class="n">users</span> <span class="o">+=</span> <span class="n">user</span><span class="o">.</span><span class="n">id</span> <span class="o">-&gt;</span> <span class="n">user</span> <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span> <span class="n">lock</span><span class="o">.</span><span class="n">writeLock</span><span class="o">().</span><span class="n">unlock</span><span class="o">()</span> <span class="o">}</span> <span class="o">}</span> <span class="k">def</span> <span class="n">findById</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="o">{</span> <span class="n">lock</span><span class="o">.</span><span class="n">readLock</span><span class="o">().</span><span class="n">lock</span><span class="o">()</span> <span class="k">try</span> <span class="o">{</span> <span class="n">users</span><span class="o">.</span><span class="n">get</span><span class="o">(</span><span class="n">id</span><span class="o">)</span> <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span> <span class="n">lock</span><span class="o">.</span><span class="n">readLock</span><span class="o">().</span><span class="n">unlock</span><span class="o">()</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>Now you think you are safe. And it does not take a couple of VCS commits for somebody (most probably you) to mess up the entire synchronization scheme:</p> <pre><code class="language-scala"><span class="c1">// Violating thread-safety on `users`.</span> <span class="k">override</span> <span class="k">def</span> <span class="n">toString</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="n">s</span><span class="s">"There are ${users.size} user(s)."</span></code></pre> <p>Oops! There can arise a lot more complicated subtle bugs. Shit can even hit the fan when you introduce multiple variables or make nested calls, that is, functions calling functions calling functions which are accessing <code>users</code>. It is obvious that you are doomed. Now good luck with your chasing the <a href="https://en.wikipedia.org/wiki/Heisenbug">Heisenbug</a> journey!</p> <p>Then it occured to me, can’t we make the compiler enforce a certain lock context while accessing to a particular set of variables? What if compiler would not allow you to read <code>users</code> if your thread did not already acquire <code>lock.readLock()</code>? Or similarly would not allow you to mutate it if you did not already acquire <code>lock.writeLock()</code>? Here is the solution that I came up with to these questions:</p> <pre><code class="language-scala"><span class="k">import</span> <span class="nn">java.util.concurrent.locks.Lock</span> <span class="k">import</span> <span class="nn">java.util.concurrent.locks.ReentrantReadWriteLock</span> <span class="k">trait</span> <span class="nc">SynchronizedAccess</span> <span class="o">{</span> <span class="k">import</span> <span class="nn">SynchronizedAccess._</span> <span class="k">protected</span> <span class="k">val</span> <span class="n">instanceLock</span><span class="k">:</span> <span class="kt">ReentrantReadWriteLock</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ReentrantReadWriteLock</span><span class="o">()</span> <span class="k">protected</span> <span class="k">val</span> <span class="n">instanceReadLock</span><span class="k">:</span> <span class="kt">ReadLock</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ReadLock</span><span class="o">(</span><span class="n">instanceLock</span><span class="o">.</span><span class="n">readLock</span><span class="o">())</span> <span class="k">protected</span> <span class="k">val</span> <span class="n">instanceReadWriteLock</span><span class="k">:</span> <span class="kt">ReadWriteLock</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ReadWriteLock</span><span class="o">(</span><span class="n">instanceLock</span><span class="o">.</span><span class="n">readLock</span><span class="o">(),</span> <span class="n">instanceLock</span><span class="o">.</span><span class="n">writeLock</span><span class="o">())</span> <span class="k">protected</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Synchronized</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="k">private</span> <span class="k">var</span> <span class="n">value</span><span class="k">:</span> <span class="kt">T</span><span class="o">)</span> <span class="o">{</span> <span class="k">def</span> <span class="n">apply</span><span class="o">()(</span><span class="k">implicit</span> <span class="n">readLock</span><span class="k">:</span> <span class="kt">ReadLock</span><span class="o">)</span><span class="k">:</span> <span class="kt">T</span> <span class="o">=</span> <span class="o">{</span> <span class="n">validateLock</span><span class="o">(</span><span class="n">readLock</span><span class="o">,</span> <span class="n">instanceReadLock</span><span class="o">,</span> <span class="n">instanceReadWriteLock</span><span class="o">)</span> <span class="n">value</span> <span class="o">}</span> <span class="k">def</span> <span class="n">update</span><span class="o">(</span><span class="n">newValue</span><span class="k">:</span> <span class="kt">T</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">readWriteLock</span><span class="k">:</span> <span class="kt">ReadWriteLock</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span> <span class="n">validateLock</span><span class="o">(</span><span class="n">readWriteLock</span><span class="o">,</span> <span class="n">instanceReadWriteLock</span><span class="o">)</span> <span class="n">value</span> <span class="k">=</span> <span class="n">newValue</span> <span class="o">}</span> <span class="k">private</span> <span class="k">def</span> <span class="n">validateLock</span><span class="o">(</span><span class="n">lock</span><span class="k">:</span> <span class="kt">TypedLock</span><span class="o">,</span> <span class="n">allowedLocks</span><span class="k">:</span> <span class="kt">TypedLock*</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span> <span class="n">require</span><span class="o">(</span><span class="n">allowedLocks</span><span class="o">.</span><span class="n">contains</span><span class="o">(</span><span class="n">lock</span><span class="o">),</span> <span class="s">"cannot be accessed from another synchronization scope"</span><span class="o">)</span> <span class="n">require</span><span class="o">(</span><span class="n">lock</span><span class="o">.</span><span class="n">tryLock</span><span class="o">(),</span> <span class="s">"cannot be accessed out of a synchronization scope"</span><span class="o">)</span> <span class="n">lock</span><span class="o">.</span><span class="n">unlock</span><span class="o">()</span> <span class="o">}</span> <span class="o">}</span> <span class="k">protected</span> <span class="k">def</span> <span class="n">synchronizeRead</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">body</span><span class="k">:</span> <span class="kt">ReadLock</span> <span class="o">=&gt;</span> <span class="n">T</span><span class="o">)</span><span class="k">:</span> <span class="kt">T</span> <span class="o">=</span> <span class="n">synchronizeOperation</span><span class="o">(</span><span class="n">instanceReadLock</span><span class="o">)(</span><span class="n">body</span><span class="o">)</span> <span class="k">protected</span> <span class="k">def</span> <span class="n">synchronizeReadWrite</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">body</span><span class="k">:</span> <span class="kt">ReadWriteLock</span> <span class="o">=&gt;</span> <span class="n">T</span><span class="o">)</span><span class="k">:</span> <span class="kt">T</span> <span class="o">=</span> <span class="n">synchronizeOperation</span><span class="o">(</span><span class="n">instanceReadWriteLock</span><span class="o">)(</span><span class="n">body</span><span class="o">)</span> <span class="k">protected</span> <span class="k">def</span> <span class="n">synchronizeOperation</span><span class="o">[</span><span class="kt">T</span>, <span class="kt">L</span> <span class="k">&lt;:</span> <span class="kt">TypedLock</span><span class="o">](</span><span class="n">lock</span><span class="k">:</span> <span class="kt">L</span><span class="o">)(</span><span class="n">body</span><span class="k">:</span> <span class="kt">L</span> <span class="o">=&gt;</span> <span class="n">T</span><span class="o">)</span><span class="k">:</span> <span class="kt">T</span> <span class="o">=</span> <span class="o">{</span> <span class="n">lock</span><span class="o">.</span><span class="n">lock</span><span class="o">()</span> <span class="k">try</span> <span class="o">{</span> <span class="n">body</span><span class="o">(</span><span class="n">lock</span><span class="o">)</span> <span class="o">}</span> <span class="k">finally</span> <span class="o">{</span> <span class="n">lock</span><span class="o">.</span><span class="n">unlock</span><span class="o">()</span> <span class="o">}</span> <span class="o">}</span> <span class="o">}</span> <span class="k">object</span> <span class="nc">SynchronizedAccess</span> <span class="o">{</span> <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">TypedLock</span> <span class="o">{</span> <span class="k">protected</span> <span class="k">val</span> <span class="n">instance</span><span class="k">:</span> <span class="kt">Lock</span> <span class="k">def</span> <span class="n">lock</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">instance</span><span class="o">.</span><span class="n">lock</span><span class="o">()</span> <span class="k">def</span> <span class="n">unlock</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">instance</span><span class="o">.</span><span class="n">unlock</span><span class="o">()</span> <span class="k">def</span> <span class="n">tryLock</span><span class="o">()</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="n">instance</span><span class="o">.</span><span class="n">tryLock</span><span class="o">()</span> <span class="o">}</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">ReadLock</span><span class="o">(</span><span class="n">readLock</span><span class="k">:</span> <span class="kt">ReentrantReadWriteLock.ReadLock</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">TypedLock</span> <span class="o">{</span> <span class="k">override</span> <span class="k">protected</span> <span class="k">val</span> <span class="n">instance</span><span class="k">:</span> <span class="kt">Lock</span> <span class="o">=</span> <span class="n">readLock</span> <span class="o">}</span> <span class="k">sealed</span> <span class="k">class</span> <span class="nc">ReadWriteLock</span> <span class="o">(</span><span class="n">readLock</span><span class="k">:</span> <span class="kt">ReentrantReadWriteLock.ReadLock</span><span class="o">,</span> <span class="n">writeLock</span><span class="k">:</span> <span class="kt">ReentrantReadWriteLock.WriteLock</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">ReadLock</span><span class="o">(</span><span class="n">readLock</span><span class="o">)</span> <span class="o">{</span> <span class="k">override</span> <span class="k">protected</span> <span class="k">val</span> <span class="n">instance</span><span class="k">:</span> <span class="kt">Lock</span> <span class="o">=</span> <span class="n">writeLock</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>Looks complicated? See me while I dance with it:</p> <pre><code class="language-scala"><span class="k">class</span> <span class="nc">UserService</span> <span class="k">extends</span> <span class="nc">SynchronizedAccess</span> <span class="o">{</span> <span class="k">private</span> <span class="k">val</span> <span class="n">users</span><span class="k">:</span> <span class="kt">Synchronized</span><span class="o">[</span><span class="kt">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">User</span><span class="o">]]</span> <span class="k">=</span> <span class="nc">Synchronized</span><span class="o">(</span><span class="nc">Map</span><span class="o">[</span><span class="kt">Int</span>, <span class="kt">User</span><span class="o">]())</span> <span class="k">private</span> <span class="k">val</span> <span class="n">managerCount</span><span class="k">:</span> <span class="kt">Synchronized</span><span class="o">[</span><span class="kt">Int</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Synchronized</span><span class="o">(</span><span class="mi">0</span><span class="o">)</span> <span class="k">def</span> <span class="n">save</span><span class="o">(</span><span class="n">user</span><span class="k">:</span> <span class="kt">User</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="c1">// Note that `users` and `managerCount` variables will be updated</span> <span class="c1">// atomically while the rest waits for the `ReadWrite` lock.</span> <span class="n">synchronizeReadWrite</span> <span class="o">{</span> <span class="k">implicit</span> <span class="n">lock</span> <span class="k">=&gt;</span> <span class="n">users</span><span class="o">()</span> <span class="o">+=</span> <span class="n">user</span><span class="o">.</span><span class="n">id</span> <span class="o">-&gt;</span> <span class="n">user</span> <span class="k">if</span> <span class="o">(</span><span class="n">user</span><span class="o">.</span><span class="n">manager</span><span class="o">)</span> <span class="n">managerCount</span><span class="o">()</span> <span class="o">+=</span> <span class="mi">1</span> <span class="o">}</span> <span class="k">def</span> <span class="n">findById</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="n">synchronizeRead</span> <span class="o">{</span> <span class="k">implicit</span> <span class="n">lock</span> <span class="k">=&gt;</span> <span class="n">users</span><span class="o">().</span><span class="n">get</span><span class="o">(</span><span class="n">id</span><span class="o">)</span> <span class="o">}</span> <span class="k">def</span> <span class="n">findAllNames</span><span class="o">()</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">synchronizeRead</span> <span class="o">{</span> <span class="k">implicit</span> <span class="n">lock</span> <span class="k">=&gt;</span> <span class="n">findAll</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">name</span><span class="o">)</span> <span class="o">}</span> <span class="c1">// Note that findAll() requires a `ReadLock` context in order to access `users`.</span> <span class="k">private</span> <span class="k">def</span> <span class="n">findAll</span><span class="o">(</span><span class="k">implicit</span> <span class="n">lock</span><span class="k">:</span> <span class="kt">ReadLock</span><span class="o">)</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="n">users</span><span class="o">().</span><span class="n">values</span><span class="o">.</span><span class="n">toSeq</span> <span class="k">def</span> <span class="n">findManagerCount</span><span class="o">()</span><span class="k">:</span> <span class="kt">Int</span> <span class="o">=</span> <span class="n">synchronizeRead</span> <span class="o">{</span> <span class="k">implicit</span> <span class="n">lock</span> <span class="k">=&gt;</span> <span class="n">managerCount</span><span class="o">()</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>In a nutshell, what did <code>SynchronizedAccess</code> trait really bring us? <em>It enforces a <strong>typed</strong> and <strong>unique</strong> locking context on the variables of type <code>Synchronized[T]</code>.</em> It is <em>typed</em> because read and read-write operations are distinct from each other in the function decleration via implicit <code>ReadLock</code> and <code>ReadWriteLock</code> parameters. It is <em>unique</em> because <code>Synchronized</code> variables can only be accessed by the instance lock inherited from <code>SynchronizedAccess</code> trait.</p> <p>Here is your free lunch. <a href="https://translate.google.com/#nl/en/eet%20smakelijk">Eet smaaklijk!</a></p> <h1 id="common-confusions">Common Confusions</h1> <p>I sadly observed that there are some common confusions about <code>Synchronized[T]</code> type. Let me try to address them here.</p> <ul> <li> <p><strong>I could have used a <code>ConcurrentMap</code> instead!</strong> <code>Map</code> usage in the examples above is just there for demonstration purposes. It does not have to be a collection at all. If you have just one variable and it is a collection, then going with a synchronized/concurrent implementation is totally fine.</p> </li> <li> <p><strong>I could have used a <code>ConcurrentMap</code> and an <code>AtomicInteger</code> instead!</strong> No, you cannot. Then you would totally spoil the <em>atomic</em> read-write operations. You will still need a <em>transaction</em>-like mechanism ala in SQL.</p> </li> </ul> tag:vy.github.io,2015-08-20://blog/post/2015/08/20/scala-futures/ Improving Performace Using Scala Futures 2015-08-20T16:20:00Z 2015-08-20T16:20:00Z <p>At <a href="http://bol.com/">bol.com</a>, we have a microservice-based software ecosystem. Every functionally grown enough monolithic software is split into its own microservice, which exposes a simple REST interface to the outside world. Fine-grained functional scope of the services helps to keep the code base clean and lets programmers just to stick to the point. Which implicitly also results in a coding style such that the <em>clean code</em> has a higher priority over performance. Optimizations are introduced whenever they are necessary and after careful profiling. For the rest, <a href="https://en.wikipedia.org/wiki/KISS_principle">KISS principle</a> is applied.</p> <h1 id="the-problem">The Problem</h1> <p>For a particular microservice, we are given the task of returning a snapshot of the current state of the data model. In this post, I will use the following simple model to demonstrate the problem at hand and the solution I came up with.</p> <p><img src="arch.png" alt="Sample service-layered architecture"></p> <p>In a nutshell, service basically encapsulates <code>User</code> and <code>Group</code> entities in the database. While doing so, database records are first interfaced via a <a href="https://en.wikipedia.org/wiki/Data_access_object">DAO</a> layer requiring a database connection to operate.</p> <pre><code class="language-scala"><span class="k">trait</span> <span class="nc">UserDao</span> <span class="o">{</span> <span class="k">def</span> <span class="n">create</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">groupId</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">connection</span><span class="k">:</span> <span class="kt">Connection</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="k">def</span> <span class="n">findAll</span><span class="o">()(</span><span class="k">implicit</span> <span class="n">connection</span><span class="k">:</span> <span class="kt">Connection</span><span class="o">)</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="o">}</span> <span class="k">trait</span> <span class="nc">GroupDao</span> <span class="o">{</span> <span class="k">def</span> <span class="n">create</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">def</span> <span class="n">findAll</span><span class="o">()(</span><span class="k">implicit</span> <span class="n">connection</span><span class="k">:</span> <span class="kt">Connection</span><span class="o">)</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">Group</span><span class="o">]</span> <span class="k">def</span> <span class="n">findById</span><span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)(</span><span class="k">implicit</span> <span class="n">connection</span><span class="k">:</span> <span class="kt">Connection</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">Group</span><span class="o">]</span> <span class="o">}</span></code></pre> <p>Next, DAO layers are interfaced by services such that they acquire a database connection from the JDBC pool and make relevant DAO calls.</p> <pre><code class="language-scala"><span class="k">trait</span> <span class="nc">UserService</span> <span class="o">{</span> <span class="k">def</span> <span class="n">create</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">groupId</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="k">def</span> <span class="n">findAll</span><span class="o">()</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="o">}</span> <span class="nd">@Component</span> <span class="k">class</span> <span class="nc">UserServiceImpl</span> <span class="nd">@Autowired</span><span class="o">()</span> <span class="o">(</span><span class="n">userDao</span><span class="k">:</span> <span class="kt">UserDao</span><span class="o">,</span> <span class="n">groupDao</span><span class="k">:</span> <span class="kt">GroupDao</span><span class="o">,</span> <span class="n">connectionPool</span><span class="k">:</span> <span class="kt">ConnectionPool</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">UserService</span> <span class="o">{</span> <span class="k">def</span> <span class="n">create</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">groupId</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">connectionPool</span><span class="o">.</span><span class="n">acquire</span> <span class="o">{</span> <span class="k">implicit</span> <span class="n">connection</span> <span class="k">=&gt;</span> <span class="n">require</span><span class="o">(</span><span class="n">groupDao</span><span class="o">.</span><span class="n">findById</span><span class="o">(</span><span class="n">groupId</span><span class="o">).</span><span class="n">isDefined</span><span class="o">,</span> <span class="s">"non-existing group id"</span><span class="o">)</span> <span class="n">userDao</span><span class="o">.</span><span class="n">create</span><span class="o">(</span><span class="n">user</span><span class="o">,</span> <span class="n">groupId</span><span class="o">)</span> <span class="o">}</span> <span class="k">def</span> <span class="n">findAll</span><span class="o">()</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">User</span><span class="o">]</span> <span class="k">=</span> <span class="n">connectionPool</span><span class="o">.</span><span class="n">acquire</span> <span class="o">{</span> <span class="k">implicit</span> <span class="n">connection</span> <span class="k">=&gt;</span> <span class="n">userDao</span><span class="o">.</span><span class="n">findAll</span><span class="o">()</span> <span class="o">}</span> <span class="o">}</span> <span class="c1">// GroupService and GroupServiceImpl are implemented likewise...</span></code></pre> <p>This also applies to <code>SnapshotService</code>, which accesses <code>UserDao</code> and <code>GroupDao</code> using a single database connection.</p> <pre><code class="language-scala"><span class="k">trait</span> <span class="nc">SnapshotService</span> <span class="o">{</span> <span class="k">def</span> <span class="n">take</span><span class="o">()</span><span class="k">:</span> <span class="kt">Snapshot</span> <span class="o">}</span> <span class="nd">@Component</span> <span class="k">class</span> <span class="nc">SnapshotServiceImpl</span> <span class="nd">@Autowired</span><span class="o">()</span> <span class="o">(</span><span class="n">userDao</span><span class="k">:</span> <span class="kt">UserDao</span><span class="o">,</span> <span class="n">groupDao</span><span class="k">:</span> <span class="kt">GroupDao</span><span class="o">,</span> <span class="n">connectionPool</span><span class="k">:</span> <span class="kt">ConnectionPool</span><span class="o">)</span> <span class="o">{</span> <span class="k">def</span> <span class="n">take</span><span class="o">()</span><span class="k">:</span> <span class="kt">Snapshot</span> <span class="o">=</span> <span class="n">connectionPool</span><span class="o">.</span><span class="n">acquire</span> <span class="o">{</span> <span class="k">implicit</span> <span class="n">connection</span> <span class="k">=&gt;</span> <span class="nc">Snapshot</span><span class="o">.</span><span class="n">from</span><span class="o">(</span> <span class="n">users</span> <span class="k">=</span> <span class="n">userDao</span><span class="o">.</span><span class="n">findAll</span><span class="o">(),</span> <span class="n">groups</span> <span class="k">=</span> <span class="n">groupDao</span><span class="o">.</span><span class="n">findAll</span><span class="o">())</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>While this approach totally makes sense, as the user and group collections start to grow in size, calls to <code>SnapshotService.take()</code> started to take more and more time.</p> <h1 id="the-solution">The Solution</h1> <p>Given the fact that constructing a snapshot is a read-only operation, I anticipated querying <code>UserDao.findAll()</code> and <code>GroupDao.findAll()</code> in parallel will buy us quite some <code>SnapshotService.take()</code> time. That being said, it is crucial to note that querying DAOs via separate database connections in parallel is going to violate the transactional safety, but in our case it has a negligible margin of error for fetching collections in parallel.</p> <p>I updated the existing <code>SnapshotService</code> to replace its <code>UserDao</code> and <code>GroupDao</code> calls to <code>UserService</code> and <code>GroupService</code> instead (to provide each DAO a dedicated connection) and perform these fetch operations in parallel through an <code>ExecutorService</code> as follows:</p> <p><img src="parallel-arch.png" alt="Sample service-layered architecture with parallelization"></p> <p>I implemented <code>ExecutorService</code> using Scala’s <code>Future</code>s:</p> <pre><code class="language-scala"><span class="k">trait</span> <span class="nc">ExecutorService</span> <span class="o">{</span> <span class="k">implicit</span> <span class="k">val</span> <span class="n">executionContext</span><span class="k">:</span> <span class="kt">ExecutionContext</span> <span class="k">val</span> <span class="n">awaitDuration</span><span class="k">:</span> <span class="kt">Duration</span> <span class="k">def</span> <span class="n">await</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">future</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span><span class="k">:</span> <span class="kt">T</span> <span class="k">def</span> <span class="n">awaitAll</span><span class="o">(</span><span class="n">futures</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="k">_</span><span class="o">]*)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="k">def</span> <span class="n">schedule</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">body</span><span class="k">:</span> <span class="o">=&gt;</span> <span class="n">T</span><span class="o">)</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">def</span> <span class="n">scheduleAll</span><span class="o">(</span><span class="n">bodies</span><span class="k">:</span> <span class="o">(()</span> <span class="o">=&gt;</span> <span class="k">_</span><span class="o">)*)</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="o">}</span> <span class="k">class</span> <span class="nc">ExecutorServiceImpl</span><span class="o">(</span><span class="n">awaitDurationSpec</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">ExecutorService</span> <span class="o">{</span> <span class="n">requireNotNull</span><span class="o">(</span><span class="s">"await duration specification"</span> <span class="o">-&gt;</span> <span class="n">awaitDurationSpec</span><span class="o">)</span> <span class="k">override</span> <span class="k">implicit</span> <span class="k">val</span> <span class="n">executionContext</span><span class="k">:</span> <span class="kt">ExecutionContext</span> <span class="o">=</span> <span class="nc">ExecutionContext</span><span class="o">.</span><span class="nc">Implicits</span><span class="o">.</span><span class="n">global</span> <span class="k">override</span> <span class="k">val</span> <span class="n">awaitDuration</span><span class="k">:</span> <span class="kt">Duration</span> <span class="o">=</span> <span class="nc">Duration</span><span class="o">(</span><span class="n">awaitDurationSpec</span><span class="o">)</span> <span class="k">override</span> <span class="k">def</span> <span class="n">await</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">future</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">T</span><span class="o">])</span><span class="k">:</span> <span class="kt">T</span> <span class="o">=</span> <span class="nc">Await</span><span class="o">.</span><span class="n">result</span><span class="o">(</span><span class="n">future</span><span class="o">,</span> <span class="n">awaitDuration</span><span class="o">)</span> <span class="k">override</span> <span class="k">def</span> <span class="n">awaitAll</span><span class="o">(</span><span class="n">futures</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="k">_</span><span class="o">]*)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="k">if</span> <span class="o">(</span><span class="n">futures</span><span class="o">.</span><span class="n">nonEmpty</span><span class="o">)</span> <span class="n">await</span> <span class="o">{</span> <span class="n">futures</span><span class="o">.</span><span class="n">reduce</span> <span class="o">{</span> <span class="o">(</span><span class="n">prev</span><span class="o">,</span> <span class="n">next</span><span class="o">)</span> <span class="k">=&gt;</span> <span class="n">prev</span><span class="o">.</span><span class="n">flatMap</span><span class="o">(</span><span class="k">_</span> <span class="k">=&gt;</span> <span class="n">next</span><span class="o">)</span> <span class="o">}</span> <span class="o">}</span> <span class="k">override</span> <span class="k">def</span> <span class="n">schedule</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">body</span><span class="k">:</span> <span class="o">=&gt;</span> <span class="n">T</span><span class="o">)</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">T</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Future</span> <span class="o">{</span> <span class="n">body</span> <span class="o">}</span> <span class="k">override</span> <span class="k">def</span> <span class="n">scheduleAll</span><span class="o">(</span><span class="n">bodies</span><span class="k">:</span> <span class="o">(()</span> <span class="o">=&gt;</span> <span class="k">_</span><span class="o">)*)</span><span class="k">:</span> <span class="kt">Future</span><span class="o">[</span><span class="kt">Unit</span><span class="o">]</span> <span class="k">=</span> <span class="k">if</span> <span class="o">(</span><span class="n">bodies</span><span class="o">.</span><span class="n">isEmpty</span><span class="o">)</span> <span class="nc">Future</span><span class="o">.</span><span class="n">successful</span><span class="o">(())</span> <span class="k">else</span> <span class="n">schedule</span> <span class="o">{</span> <span class="n">bodies</span><span class="o">.</span><span class="n">map</span><span class="o">(</span><span class="n">body</span> <span class="k">=&gt;</span> <span class="n">schedule</span> <span class="o">{</span> <span class="n">body</span><span class="o">()</span> <span class="o">}).</span><span class="n">foreach</span><span class="o">(</span><span class="n">await</span><span class="o">)</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>And then employing <code>ExecutorService</code> in <code>SnapshotService</code> is a piece of cake:</p> <pre><code class="language-scala"><span class="nd">@Component</span> <span class="k">class</span> <span class="nc">SnapshotServiceImpl</span> <span class="nd">@Autowired</span><span class="o">()</span> <span class="o">(</span><span class="n">userService</span><span class="k">:</span> <span class="kt">UserService</span><span class="o">,</span> <span class="n">groupService</span><span class="k">:</span> <span class="kt">GroupService</span><span class="o">,</span> <span class="n">executorService</span><span class="k">:</span> <span class="kt">ExecutorService</span><span class="o">)</span> <span class="o">{</span> <span class="k">import</span> <span class="nn">executorService._</span> <span class="k">def</span> <span class="n">take</span><span class="o">()</span><span class="k">:</span> <span class="kt">Snapshot</span> <span class="o">=</span> <span class="o">{</span> <span class="k">val</span> <span class="n">usersFuture</span> <span class="k">=</span> <span class="n">schedule</span> <span class="o">{</span> <span class="n">userService</span><span class="o">.</span><span class="n">findAll</span><span class="o">()</span> <span class="o">}</span> <span class="k">val</span> <span class="n">groupsFuture</span> <span class="k">=</span> <span class="n">schedule</span> <span class="o">{</span> <span class="n">groupService</span><span class="o">.</span><span class="n">findAll</span><span class="o">()</span> <span class="o">}</span> <span class="n">await</span> <span class="o">{</span> <span class="k">for</span> <span class="o">{</span> <span class="n">users</span> <span class="k">&lt;-</span> <span class="n">usersFuture</span> <span class="n">groups</span> <span class="k">&lt;-</span> <span class="n">groupsFuture</span> <span class="o">}</span> <span class="k">yield</span> <span class="nc">Snapshot</span><span class="o">.</span><span class="n">from</span><span class="o">(</span><span class="n">users</span><span class="o">,</span> <span class="n">groups</span><span class="o">)</span> <span class="o">}</span> <span class="o">}</span> <span class="o">}</span></code></pre> <h1 id="conclusion">Conclusion</h1> <p><a href="http://www.scala-lang.org/api/current/index.html#scala.concurrent.Future">Scala Future</a>s provide a powerful abstraction to run certain tasks in parallel using a thread pool in the back of the scenes. At the beginning I was afraid that it might necessitate too much boilerplate just to get a simple thing done, but luckily it turned out to be a quite easy task.</p> <p>In this example, I needed to trade off the transactional consistency of fetched collections for the sake of increasing performance using multiple threads to query the database. While this implied a negligible error margin for the particular problem at hand here, it is not something desirable at every microservice. But that is the discussion of another blog post.</p> tag:vy.github.io,2015-08-15://blog/post/2015/08/15/vaadin-scala-shortcuts/ Scala Shortcuts for Vaadin Development 2015-08-15T15:25:00Z 2015-08-15T15:25:00Z <p>This year I had found chance to get my hands dirty with <a href="https://vaadin.com/">Vaadin</a>. Given the fact that this was the first time I was exposed to a GWT-based web framework using Scala, I tried various coding conventions, utility functions, shortcuts, etc. in the beginning. Over the time, I came up with some common conventions that I employ throughout the code base. In this post, I will share some of these custom tricks I developed along this pursuit.</p> <h1 id="returning-the-first-value">“Returning” the First Value</h1> <p>While creating a certain UI component, what a programmer occasionally performs is to 1) instantiate the class, 2) set certain properties, and 3) return/use the instance.</p> <pre><code class="language-scala"><span class="k">val</span> <span class="n">buttonLayout</span><span class="k">:</span> <span class="kt">HorizontalLayout</span> <span class="o">=</span> <span class="o">{</span> <span class="k">val</span> <span class="n">layout</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">HorizontalLayout</span> <span class="n">layout</span><span class="o">.</span><span class="n">setMargin</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="n">layout</span><span class="o">.</span><span class="n">setSpacing</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="n">layout</span><span class="o">.</span><span class="n">addComponent</span><span class="o">(</span><span class="n">submitButton</span><span class="o">)</span> <span class="n">layout</span><span class="o">.</span><span class="n">addComponent</span><span class="o">(</span><span class="n">resetButton</span><span class="o">)</span> <span class="n">layout</span> <span class="o">}</span></code></pre> <p>This pattern was so idiomatic and repetitive throughout the code base that I thought having something similar to <a href="http://www.lispworks.com/documentation/HyperSpec/Body/m_prog1c.htm">prog1 in Common Lisp</a> would be really helpful for structuring similar components. Hence, I came up with my own <code>returning</code> utility function as follows:</p> <pre><code class="language-scala"><span class="k">def</span> <span class="n">returning</span><span class="o">[</span><span class="kt">T</span><span class="o">](</span><span class="n">result</span><span class="k">:</span> <span class="kt">T</span><span class="o">)(</span><span class="n">body</span><span class="k">:</span> <span class="kt">T</span> <span class="o">=&gt;</span> <span class="nc">Unit</span><span class="o">)</span><span class="k">:</span> <span class="kt">T</span> <span class="o">=</span> <span class="o">{</span> <span class="n">body</span><span class="o">(</span><span class="n">result</span><span class="o">)</span> <span class="n">result</span> <span class="o">}</span></code></pre> <p>When we employ <code>returning</code> in <code>buttonLayout</code>, code translates as follows:</p> <pre><code class="language-scala"><span class="k">val</span> <span class="n">buttonLayout</span><span class="k">:</span> <span class="kt">HorizontalLayout</span> <span class="o">=</span> <span class="n">returning</span><span class="o">(</span><span class="k">new</span> <span class="nc">HorizontalLayout</span><span class="o">)</span> <span class="o">{</span> <span class="n">layout</span> <span class="k">=&gt;</span> <span class="n">layout</span><span class="o">.</span><span class="n">setMargin</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="n">layout</span><span class="o">.</span><span class="n">setSpacing</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="n">layout</span><span class="o">.</span><span class="n">addComponent</span><span class="o">(</span><span class="n">submitButton</span><span class="o">)</span> <span class="n">layout</span><span class="o">.</span><span class="n">addComponent</span><span class="o">(</span><span class="n">resetButton</span><span class="o">)</span> <span class="o">}</span></code></pre> <p>I find this version more clear on intent.</p> <h1 id="reading-nullable-text-fields">Reading Nullable Text Fields</h1> <p>While coding in Scala, I write my code as if there are no <code>null</code>s. And whenever there is an external API that I need to communicate and has potential to return <code>null</code>, I wrap it in an <code>Option</code>. This practice also applies to <code>getValue</code> method of <code>TextField</code>s in Vaadin. Hence, I purposed a common function to read from text fields:</p> <pre><code class="language-scala"><span class="k">def</span> <span class="n">getTrimmedValue</span><span class="o">(</span><span class="n">field</span><span class="k">:</span> <span class="kt">AbstractTextField</span><span class="o">)</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Option</span><span class="o">(</span><span class="n">field</span><span class="o">.</span><span class="n">getValue</span><span class="o">).</span><span class="n">filterNot</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">trim</span><span class="o">.</span><span class="n">isEmpty</span><span class="o">)</span></code></pre> <p>Note that <code>getTrimmedValue</code> treats input fields of redundant whitespace as a <code>None</code>.</p> <h1 id="grouping-component-getterssetters">Grouping Component Getters/Setters</h1> <p>While creating forms, I was using the following scheme to implement fields one by one:</p> <pre><code class="language-scala"><span class="k">val</span> <span class="n">nameField</span><span class="k">:</span> <span class="kt">TextField</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">TextField</span> <span class="k">def</span> <span class="n">getNameFieldValue</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">getTrimmedValue</span><span class="o">(</span><span class="n">nameField</span><span class="o">)</span> <span class="k">def</span> <span class="n">resetNameFieldValue</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">nameField</span><span class="o">.</span><span class="n">setValue</span><span class="o">(</span><span class="n">name</span><span class="o">.</span><span class="n">orNull</span><span class="o">)</span> <span class="k">val</span> <span class="n">surnameField</span><span class="k">:</span> <span class="kt">TextField</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">TextField</span> <span class="k">def</span> <span class="n">getSurnameFieldValue</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="n">getTrimmedValue</span><span class="o">(</span><span class="n">surnameField</span><span class="o">)</span> <span class="k">def</span> <span class="n">resetSurameFieldValue</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">surnameField</span><span class="o">.</span><span class="n">setValue</span><span class="o">(</span><span class="n">surname</span><span class="o">.</span><span class="n">orNull</span><span class="o">)</span> <span class="c1">// ...</span> <span class="k">def</span> <span class="n">reset</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span> <span class="n">resetNameFieldValue</span><span class="o">()</span> <span class="n">resetSurnameFieldValue</span><span class="o">()</span> <span class="c1">// ...</span> <span class="o">}</span></code></pre> <p>Per see, the namespace of the class gets polluted as you add more fields. That is, in order to implement a form of 10 fields, you end up with at least 3x10=30 class variables/methods. In order to mitigate this problem, I wrote a custom trait to group component accessors into a single field:</p> <pre><code class="language-scala"><span class="k">trait</span> <span class="nc">CustomField</span><span class="o">[</span><span class="kt">ComponentType</span> <span class="k">&lt;:</span> <span class="kt">Component</span>, <span class="kt">ValueType</span><span class="o">]</span> <span class="o">{</span> <span class="k">val</span> <span class="n">component</span><span class="k">:</span> <span class="kt">ComponentType</span> <span class="k">def</span> <span class="n">value</span><span class="o">()</span><span class="k">:</span> <span class="kt">ValueType</span> <span class="k">def</span> <span class="n">reset</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">}</span></code></pre> <p>Using <code>CustomField</code>, the above code translates to this:</p> <pre><code class="language-scala"><span class="k">val</span> <span class="n">nameField</span><span class="k">:</span> <span class="kt">CustomField</span><span class="o">[</span><span class="kt">TextField</span>, <span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">CustomField</span><span class="o">[</span><span class="kt">TextField</span>, <span class="kt">String</span><span class="o">]</span> <span class="o">{</span> <span class="k">val</span> <span class="n">component</span><span class="k">:</span> <span class="kt">TextField</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">TextField</span> <span class="k">def</span> <span class="n">value</span><span class="o">()</span><span class="k">:</span> <span class="kt">ComponentHelpers.getTrimmedValue</span><span class="o">(</span><span class="kt">component</span><span class="o">)</span> <span class="kt">def</span> <span class="kt">reset</span><span class="o">()</span><span class="kt">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">nameField</span><span class="o">.</span><span class="n">setValue</span><span class="o">(</span><span class="n">name</span><span class="o">.</span><span class="n">orNull</span><span class="o">)</span> <span class="o">}</span> <span class="k">val</span> <span class="n">surnameField</span><span class="k">:</span> <span class="kt">CustomField</span><span class="o">[</span><span class="kt">TextField</span>, <span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">CustomField</span><span class="o">[</span><span class="kt">TextField</span>, <span class="kt">String</span><span class="o">]</span> <span class="o">{</span> <span class="k">val</span> <span class="n">component</span><span class="k">:</span> <span class="kt">TextField</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">TextField</span> <span class="k">def</span> <span class="n">value</span><span class="o">()</span><span class="k">:</span> <span class="kt">ComponentHelpers.getTrimmedValue</span><span class="o">(</span><span class="kt">component</span><span class="o">)</span> <span class="kt">def</span> <span class="kt">reset</span><span class="o">()</span><span class="kt">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">surnameField</span><span class="o">.</span><span class="n">setValue</span><span class="o">(</span><span class="n">surname</span><span class="o">.</span><span class="n">orNull</span><span class="o">)</span> <span class="o">}</span> <span class="c1">// ...</span> <span class="k">val</span> <span class="n">fields</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">CustomField</span><span class="o">[</span><span class="k">_</span> <span class="k">&lt;:</span> <span class="kt">Component</span>, <span class="k">_</span><span class="o">]]</span> <span class="k">=</span> <span class="nc">Seq</span><span class="o">(</span> <span class="n">nameField</span><span class="o">,</span> <span class="n">surnameField</span><span class="o">,</span> <span class="c1">// ...</span> <span class="o">)</span> <span class="k">def</span> <span class="n">reset</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">fields</span><span class="o">.</span><span class="n">foreach</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">reset</span><span class="o">())</span></code></pre> <h1 id="confirmation-dialogs">Confirmation Dialogs</h1> <p>This one needs no introduction I guess. Here it is:</p> <pre><code class="language-scala"><span class="k">def</span> <span class="n">confirmationDialog</span><span class="o">(</span><span class="n">title</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">content</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">ok</span><span class="k">:</span> <span class="o">()</span> <span class="o">=&gt;</span> <span class="nc">Unit</span><span class="o">,</span> <span class="n">cancel</span><span class="k">:</span> <span class="o">()</span> <span class="o">=&gt;</span> <span class="nc">Unit</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span> <span class="k">lazy</span> <span class="k">val</span> <span class="n">dialog</span><span class="k">:</span> <span class="kt">ConfirmationDialog</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ConfirmationDialog</span><span class="o">(</span> <span class="n">title</span><span class="o">,</span> <span class="n">content</span><span class="o">,</span> <span class="k">new</span> <span class="nc">ClickButtonEventHandler</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">handleEvent</span><span class="o">(</span><span class="n">event</span><span class="k">:</span> <span class="kt">ClickEvent</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="n">ok</span><span class="o">()</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">{</span> <span class="k">case</span> <span class="k">_:</span> <span class="kt">Throwable</span> <span class="o">=&gt;</span> <span class="n">dialog</span><span class="o">.</span><span class="n">close</span><span class="o">()</span> <span class="o">}</span> <span class="o">}</span> <span class="o">},</span> <span class="k">new</span> <span class="nc">ClickButtonEventHandler</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">handleEvent</span><span class="o">(</span><span class="n">event</span><span class="k">:</span> <span class="kt">ClickEvent</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="n">cancel</span><span class="o">()</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">{</span> <span class="k">case</span> <span class="k">_:</span> <span class="kt">Throwable</span> <span class="o">=&gt;</span> <span class="n">dialog</span><span class="o">.</span><span class="n">close</span><span class="o">()</span> <span class="o">}</span> <span class="o">}</span> <span class="o">})</span> <span class="nc">UI</span><span class="o">.</span><span class="n">getCurrent</span><span class="o">.</span><span class="n">addWindow</span><span class="o">(</span><span class="n">dialog</span><span class="o">)</span> <span class="o">}</span></code></pre> <p>And you use it as follows:</p> <pre><code class="language-scala"><span class="k">override</span> <span class="k">def</span> <span class="n">uploadFinished</span><span class="o">(</span><span class="n">event</span><span class="k">:</span> <span class="kt">FinishedEvent</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">confirmationDialog</span><span class="o">(</span> <span class="s">"Deploy Shelf Plan"</span><span class="o">,</span> <span class="s">"""You are about to populate tables using the provided shelf plan.&lt;br/&gt;</span> <span class="s"> |Do you want to proceed?"""</span><span class="o">.</span><span class="n">stripMargin</span><span class="o">,</span> <span class="n">ok</span><span class="o">,</span> <span class="n">cancel</span><span class="o">)</span></code></pre> <h1 id="escapeable-window">“Escapeable” Window</h1> <p>So you have a <code>Window</code>, that you want to be <code>closeable</code> using the <code>ESCAPE</code> key. Fine, just extend from <code>EscapeableWindow</code>:</p> <pre><code class="language-scala"><span class="k">trait</span> <span class="nc">EscapeableWindow</span> <span class="o">{</span> <span class="n">self</span><span class="k">:</span> <span class="kt">Window</span> <span class="o">=&gt;</span> <span class="k">protected</span> <span class="k">val</span> <span class="n">escapeActionHandler</span><span class="k">:</span> <span class="kt">Handler</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Handler</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">handleAction</span><span class="o">(</span><span class="n">action</span><span class="k">:</span> <span class="kt">Action</span><span class="o">,</span> <span class="n">sender</span><span class="k">:</span> <span class="kt">scala.Any</span><span class="o">,</span> <span class="n">target</span><span class="k">:</span> <span class="kt">scala.Any</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="k">if</span> <span class="o">(</span><span class="n">action</span><span class="o">.</span><span class="n">equals</span><span class="o">(</span><span class="nc">EscapeableWindow</span><span class="o">.</span><span class="n">escapeAction</span><span class="o">))</span> <span class="n">close</span><span class="o">()</span> <span class="k">override</span> <span class="k">def</span> <span class="n">getActions</span><span class="o">(</span><span class="n">target</span><span class="k">:</span> <span class="kt">scala.Any</span><span class="o">,</span> <span class="n">sender</span><span class="k">:</span> <span class="kt">scala.Any</span><span class="o">)</span><span class="k">:</span> <span class="kt">Array</span><span class="o">[</span><span class="kt">Action</span><span class="o">]</span> <span class="k">=</span> <span class="nc">Array</span><span class="o">(</span><span class="nc">EscapeableWindow</span><span class="o">.</span><span class="n">escapeAction</span><span class="o">)</span> <span class="o">}</span> <span class="n">setClosable</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="k">this</span><span class="o">.</span><span class="n">addActionHandler</span><span class="o">(</span><span class="n">escapeActionHandler</span><span class="o">)</span> <span class="o">}</span> <span class="k">object</span> <span class="nc">EscapeableWindow</span> <span class="o">{</span> <span class="k">val</span> <span class="n">escapeAction</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">ShortcutAction</span><span class="o">(</span><span class="s">"ESCAPE"</span><span class="o">,</span> <span class="nc">ShortcutAction</span><span class="o">.</span><span class="nc">KeyCode</span><span class="o">.</span><span class="nc">ESCAPE</span><span class="o">,</span> <span class="kc">null</span><span class="o">)</span> <span class="o">}</span></code></pre> <h1 id="boolean-combo-box">Boolean Combo Box</h1> <p>For plain boolean options, you can just simply go with a check box. But if you have a nullable boolean field, you need a representation to encapsulate three different states: 1) true, 2) false, and 3) null. For that purpose, I use a combo box as follows:</p> <pre><code class="language-scala"><span class="k">def</span> <span class="n">createBooleanComboBox</span> <span class="o">(</span><span class="n">id</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">caption</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">description</span><span class="k">:</span> <span class="kt">Option</span><span class="o">[</span><span class="kt">String</span><span class="o">]</span> <span class="k">=</span> <span class="nc">None</span><span class="o">,</span> <span class="n">nullSelectionAllowed</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="kc">false</span><span class="o">)</span><span class="k">:</span> <span class="kt">ComboBox</span> <span class="o">=</span> <span class="nc">Commons</span><span class="o">.</span><span class="n">returning</span><span class="o">(</span><span class="k">new</span> <span class="nc">ComboBox</span><span class="o">(</span><span class="n">id</span><span class="o">,</span> <span class="n">caption</span><span class="o">))</span> <span class="o">{</span> <span class="n">field</span> <span class="k">=&gt;</span> <span class="n">field</span><span class="o">.</span><span class="n">setNullSelectionAllowed</span><span class="o">(</span><span class="n">nullSelectionAllowed</span><span class="o">)</span> <span class="n">description</span><span class="o">.</span><span class="n">foreach</span><span class="o">(</span><span class="n">field</span><span class="o">.</span><span class="n">setDescription</span><span class="o">)</span> <span class="nc">Seq</span><span class="o">(</span><span class="kc">true</span><span class="o">,</span> <span class="kc">false</span><span class="o">).</span><span class="n">foreach</span><span class="o">(</span><span class="n">field</span><span class="o">.</span><span class="n">addItem</span><span class="o">)</span> <span class="o">}</span></code></pre> <p>And below you can find a field that uses <code>createBooleanComboBox</code>:</p> <pre><code class="language-scala"><span class="k">val</span> <span class="n">outputMultiValueField</span><span class="k">:</span> <span class="kt">ComboBox</span> <span class="o">=</span> <span class="n">createBooleanComboBox</span><span class="o">(</span> <span class="s">"outputMultiValue"</span><span class="o">,</span> <span class="s">"Output Multi-Value"</span><span class="o">,</span> <span class="n">nullSelectionAllowed</span> <span class="k">=</span> <span class="kc">true</span><span class="o">)</span></code></pre> <h1 id="tables">Tables</h1> <p>When I first started working with tables in Vaadin, I – as probably everybody else in this business – felt the urge to abstract away the repetitive patterns while creating a table. And I came up with the following <code>CustomTableHeader</code> and <code>TableHelpers</code> utility classes.</p> <pre><code class="language-scala"><span class="k">import</span> <span class="nn">java.lang.</span><span class="o">{</span><span class="nc">Boolean</span> <span class="k">=&gt;</span> <span class="nc">JBoolean</span><span class="o">}</span> <span class="k">import</span> <span class="nn">java.lang.</span><span class="o">{</span><span class="nc">Long</span> <span class="k">=&gt;</span> <span class="nc">JLong</span><span class="o">}</span> <span class="k">import</span> <span class="nn">java.util.Locale</span> <span class="k">import</span> <span class="nn">com.bol.vaadin.common.ui.StringToLongConverter</span> <span class="k">import</span> <span class="nn">com.vaadin.data.util.converter.</span><span class="o">{</span><span class="nc">Converter</span> <span class="k">=&gt;</span> <span class="nc">VConverter</span><span class="o">}</span> <span class="k">import</span> <span class="nn">com.vaadin.data.util.converter.StringToIntegerConverter</span> <span class="k">import</span> <span class="nn">com.vaadin.ui.Table</span> <span class="k">import</span> <span class="nn">com.vaadin.ui.Table.Align</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">CustomTableHeader</span> <span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">clazz</span><span class="k">:</span> <span class="kt">Class</span><span class="o">[</span><span class="k">_</span><span class="o">],</span> <span class="n">configurations</span><span class="k">:</span> <span class="kt">Set</span><span class="o">[</span><span class="kt">CustomTableHeader.Configuration</span><span class="o">])</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">toString</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="n">name</span> <span class="o">}</span> <span class="k">object</span> <span class="nc">CustomTableHeader</span> <span class="o">{</span> <span class="k">def</span> <span class="n">apply</span> <span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">clazz</span><span class="k">:</span> <span class="kt">Class</span><span class="o">[</span><span class="k">_</span><span class="o">],</span> <span class="n">configurations</span><span class="k">:</span> <span class="kt">CustomTableHeader.Configuration*</span><span class="o">)</span><span class="k">:</span> <span class="kt">CustomTableHeader</span> <span class="o">=</span> <span class="nc">CustomTableHeader</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">clazz</span><span class="o">,</span> <span class="n">configurations</span><span class="o">.</span><span class="n">toSet</span><span class="o">)</span> <span class="k">sealed</span> <span class="k">trait</span> <span class="nc">Configuration</span> <span class="o">{</span> <span class="k">def</span> <span class="n">configure</span><span class="o">(</span><span class="n">table</span><span class="k">:</span> <span class="kt">Table</span><span class="o">,</span> <span class="n">propertyId</span><span class="k">:</span> <span class="kt">AnyRef</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">}</span> <span class="k">object</span> <span class="nc">Configuration</span> <span class="o">{</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">ExpandRatio</span><span class="o">(</span><span class="n">expandRatio</span><span class="k">:</span> <span class="kt">Float</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">Configuration</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">configure</span><span class="o">(</span><span class="n">table</span><span class="k">:</span> <span class="kt">Table</span><span class="o">,</span> <span class="n">propertyId</span><span class="k">:</span> <span class="kt">AnyRef</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">setColumnExpandRatio</span><span class="o">(</span><span class="n">propertyId</span><span class="o">,</span> <span class="n">expandRatio</span><span class="o">)</span> <span class="o">}</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Alignment</span><span class="o">(</span><span class="n">alignment</span><span class="k">:</span> <span class="kt">Align</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">Configuration</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">configure</span><span class="o">(</span><span class="n">table</span><span class="k">:</span> <span class="kt">Table</span><span class="o">,</span> <span class="n">propertyId</span><span class="k">:</span> <span class="kt">AnyRef</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">setColumnAlignment</span><span class="o">(</span><span class="n">propertyId</span><span class="o">,</span> <span class="n">alignment</span><span class="o">)</span> <span class="o">}</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Converter</span><span class="o">(</span><span class="n">converter</span><span class="k">:</span> <span class="kt">VConverter</span><span class="o">[</span><span class="kt">String</span>, <span class="k">_</span><span class="o">])</span> <span class="k">extends</span> <span class="nc">Configuration</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">configure</span><span class="o">(</span><span class="n">table</span><span class="k">:</span> <span class="kt">Table</span><span class="o">,</span> <span class="n">propertyId</span><span class="k">:</span> <span class="kt">AnyRef</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">setConverter</span><span class="o">(</span><span class="n">propertyId</span><span class="o">,</span> <span class="n">converter</span><span class="o">)</span> <span class="o">}</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Collapsed</span><span class="o">(</span><span class="n">collapsed</span><span class="k">:</span> <span class="kt">Boolean</span><span class="o">)</span> <span class="k">extends</span> <span class="nc">Configuration</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">configure</span><span class="o">(</span><span class="n">table</span><span class="k">:</span> <span class="kt">Table</span><span class="o">,</span> <span class="n">propertyId</span><span class="k">:</span> <span class="kt">AnyRef</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="k">if</span> <span class="o">(</span><span class="n">table</span><span class="o">.</span><span class="n">isColumnCollapsingAllowed</span><span class="o">)</span> <span class="n">table</span><span class="o">.</span><span class="n">setColumnCollapsed</span><span class="o">(</span><span class="n">propertyId</span><span class="o">,</span> <span class="n">collapsed</span><span class="o">)</span> <span class="o">}</span> <span class="o">}</span> <span class="k">private</span> <span class="k">lazy</span> <span class="k">val</span> <span class="n">longToStringConverter</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">StringToLongConverter</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">convertToPresentation</span><span class="o">(</span><span class="n">value</span><span class="k">:</span> <span class="kt">JLong</span><span class="o">,</span> <span class="n">locale</span><span class="k">:</span> <span class="kt">Locale</span><span class="o">)</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="nc">Option</span><span class="o">(</span><span class="n">value</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">toString</span><span class="o">).</span><span class="n">orNull</span> <span class="o">}</span> <span class="k">private</span> <span class="k">lazy</span> <span class="k">val</span> <span class="n">intToStringConvert</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">StringToIntegerConverter</span> <span class="o">{</span> <span class="k">override</span> <span class="k">def</span> <span class="n">convertToPresentation</span><span class="o">(</span><span class="n">value</span><span class="k">:</span> <span class="kt">Integer</span><span class="o">,</span> <span class="n">locale</span><span class="k">:</span> <span class="kt">Locale</span><span class="o">)</span><span class="k">:</span> <span class="kt">String</span> <span class="o">=</span> <span class="nc">Option</span><span class="o">(</span><span class="n">value</span><span class="o">).</span><span class="n">map</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">toString</span><span class="o">).</span><span class="n">orNull</span> <span class="o">}</span> <span class="k">sealed</span> <span class="k">case</span> <span class="k">class</span> <span class="nc">Builder</span><span class="o">(</span><span class="k">private</span> <span class="k">val</span> <span class="n">headers</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">CustomTableHeader</span><span class="o">])</span> <span class="o">{</span> <span class="k">def</span> <span class="n">build</span><span class="o">()</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">CustomTableHeader</span><span class="o">]</span> <span class="k">=</span> <span class="n">headers</span> <span class="k">def</span> <span class="n">add</span><span class="o">(</span><span class="n">header</span><span class="k">:</span> <span class="kt">CustomTableHeader</span><span class="o">)</span><span class="k">:</span> <span class="kt">Builder</span> <span class="o">=</span> <span class="nc">Builder</span><span class="o">(</span><span class="n">headers</span> <span class="o">:+</span> <span class="n">header</span><span class="o">)</span> <span class="k">def</span> <span class="n">addBoolean</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">configurations</span><span class="k">:</span> <span class="kt">CustomTableHeader.Configuration*</span><span class="o">)</span><span class="k">:</span> <span class="kt">Builder</span> <span class="o">=</span> <span class="n">add</span><span class="o">(</span><span class="nc">CustomTableHeader</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">classOf</span><span class="o">[</span><span class="kt">JBoolean</span><span class="o">],</span> <span class="n">configurations</span><span class="k">:</span> <span class="k">_</span><span class="kt">*</span><span class="o">))</span> <span class="k">def</span> <span class="n">addInt</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">configurations</span><span class="k">:</span> <span class="kt">CustomTableHeader.Configuration*</span><span class="o">)</span><span class="k">:</span> <span class="kt">Builder</span> <span class="o">=</span> <span class="o">{</span> <span class="k">val</span> <span class="n">extendedConfigurations</span> <span class="k">=</span> <span class="n">configurations</span> <span class="o">:+</span> <span class="nc">Configuration</span><span class="o">.</span><span class="nc">Alignment</span><span class="o">(</span><span class="nc">Table</span><span class="o">.</span><span class="nc">Align</span><span class="o">.</span><span class="nc">RIGHT</span><span class="o">)</span> <span class="o">:+</span> <span class="nc">Configuration</span><span class="o">.</span><span class="nc">Converter</span><span class="o">(</span><span class="n">intToStringConvert</span><span class="o">)</span> <span class="n">add</span><span class="o">(</span><span class="nc">CustomTableHeader</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">classOf</span><span class="o">[</span><span class="kt">Integer</span><span class="o">],</span> <span class="n">extendedConfigurations</span><span class="k">:</span> <span class="k">_</span><span class="kt">*</span><span class="o">))</span> <span class="o">}</span> <span class="k">def</span> <span class="n">addLong</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">configurations</span><span class="k">:</span> <span class="kt">CustomTableHeader.Configuration*</span><span class="o">)</span><span class="k">:</span> <span class="kt">Builder</span> <span class="o">=</span> <span class="o">{</span> <span class="k">val</span> <span class="n">extendedConfigurations</span> <span class="k">=</span> <span class="n">configurations</span> <span class="o">:+</span> <span class="nc">Configuration</span><span class="o">.</span><span class="nc">Alignment</span><span class="o">(</span><span class="nc">Table</span><span class="o">.</span><span class="nc">Align</span><span class="o">.</span><span class="nc">RIGHT</span><span class="o">)</span> <span class="o">:+</span> <span class="nc">Configuration</span><span class="o">.</span><span class="nc">Converter</span><span class="o">(</span><span class="n">longToStringConverter</span><span class="o">)</span> <span class="n">add</span><span class="o">(</span><span class="nc">CustomTableHeader</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">classOf</span><span class="o">[</span><span class="kt">Integer</span><span class="o">],</span> <span class="n">extendedConfigurations</span><span class="k">:</span> <span class="k">_</span><span class="kt">*</span><span class="o">))</span> <span class="o">}</span> <span class="k">def</span> <span class="n">addString</span><span class="o">(</span><span class="n">name</span><span class="k">:</span> <span class="kt">String</span><span class="o">,</span> <span class="n">configurations</span><span class="k">:</span> <span class="kt">CustomTableHeader.Configuration*</span><span class="o">)</span><span class="k">:</span> <span class="kt">Builder</span> <span class="o">=</span> <span class="n">add</span><span class="o">(</span><span class="nc">CustomTableHeader</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">classOf</span><span class="o">[</span><span class="kt">String</span><span class="o">],</span> <span class="n">configurations</span><span class="k">:</span> <span class="k">_</span><span class="kt">*</span><span class="o">))</span> <span class="o">}</span> <span class="k">def</span> <span class="n">builder</span><span class="o">()</span><span class="k">:</span> <span class="kt">Builder</span> <span class="o">=</span> <span class="nc">Builder</span><span class="o">(</span><span class="nc">Seq</span><span class="o">())</span> <span class="o">}</span> <span class="k">object</span> <span class="nc">TableHelpers</span> <span class="o">{</span> <span class="k">def</span> <span class="n">setHeaders</span><span class="o">(</span><span class="n">table</span><span class="k">:</span> <span class="kt">Table</span><span class="o">,</span> <span class="n">headers</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">CustomTableHeader</span><span class="o">])</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="n">headers</span><span class="o">.</span><span class="n">foreach</span> <span class="o">{</span> <span class="n">header</span> <span class="k">=&gt;</span> <span class="n">table</span><span class="o">.</span><span class="n">addContainerProperty</span><span class="o">(</span><span class="n">header</span><span class="o">.</span><span class="n">name</span><span class="o">,</span> <span class="n">header</span><span class="o">.</span><span class="n">clazz</span><span class="o">,</span> <span class="kc">null</span><span class="o">)</span> <span class="n">header</span><span class="o">.</span><span class="n">configurations</span><span class="o">.</span><span class="n">foreach</span><span class="o">(</span><span class="k">_</span><span class="o">.</span><span class="n">configure</span><span class="o">(</span><span class="n">table</span><span class="o">,</span> <span class="n">header</span><span class="o">.</span><span class="n">name</span><span class="o">))</span> <span class="o">}</span> <span class="o">}</span></code></pre> <p>Then I enjoyed this abstraction throughout all the tables I created from then on:</p> <pre><code class="language-scala"><span class="k">val</span> <span class="n">table</span><span class="k">:</span> <span class="kt">TreeTable</span> <span class="o">=</span> <span class="o">{</span> <span class="k">val</span> <span class="n">headers</span><span class="k">:</span> <span class="kt">Seq</span><span class="o">[</span><span class="kt">CustomTableHeader</span><span class="o">]</span> <span class="k">=</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="n">builder</span><span class="o">()</span> <span class="o">.</span><span class="n">addString</span><span class="o">(</span> <span class="s">"Property Name"</span><span class="o">,</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="nc">ExpandRatio</span><span class="o">(</span><span class="mi">6</span><span class="o">))</span> <span class="o">.</span><span class="n">add</span><span class="o">(</span><span class="nc">CustomTableHeader</span><span class="o">(</span> <span class="s">"Shop Code"</span><span class="o">,</span> <span class="n">classOf</span><span class="o">[</span><span class="kt">MappingShopCode</span><span class="o">],</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="nc">ExpandRatio</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span> <span class="o">.</span><span class="n">addBoolean</span><span class="o">(</span> <span class="s">"Content?"</span><span class="o">,</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="nc">Collapsed</span><span class="o">(</span><span class="kc">true</span><span class="o">),</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="nc">ExpandRatio</span><span class="o">(</span><span class="mi">1</span><span class="o">))</span> <span class="o">.</span><span class="n">addBoolean</span><span class="o">(</span> <span class="s">"Forge?"</span><span class="o">,</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="nc">Collapsed</span><span class="o">(</span><span class="kc">true</span><span class="o">),</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="nc">ExpandRatio</span><span class="o">(</span><span class="mi">1</span><span class="o">))</span> <span class="o">.</span><span class="n">add</span><span class="o">(</span><span class="nc">CustomTableHeader</span><span class="o">(</span> <span class="s">"Inp. Type"</span><span class="o">,</span> <span class="n">classOf</span><span class="o">[</span><span class="kt">MappingTypeCode</span><span class="o">],</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="nc">Collapsed</span><span class="o">(</span><span class="kc">true</span><span class="o">),</span> <span class="nc">CustomTableHeader</span><span class="o">.</span><span class="nc">Configuration</span><span class="o">.</span><span class="nc">ExpandRatio</span><span class="o">(</span><span class="mi">1</span><span class="o">)))</span> <span class="o">.</span><span class="n">build</span><span class="o">()</span> <span class="k">def</span> <span class="n">setBody</span><span class="o">(</span><span class="n">table</span><span class="k">:</span> <span class="kt">TreeTable</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span> <span class="o">=</span> <span class="o">???</span> <span class="n">returning</span><span class="o">(</span><span class="k">new</span> <span class="nc">TreeTable</span><span class="o">)</span> <span class="o">{</span> <span class="n">table</span> <span class="k">=&gt;</span> <span class="n">table</span><span class="o">.</span><span class="n">setSelectable</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="n">table</span><span class="o">.</span><span class="n">setSortEnabled</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="n">table</span><span class="o">.</span><span class="n">setColumnCollapsingAllowed</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="nc">TableHelpers</span><span class="o">.</span><span class="n">setHeaders</span><span class="o">(</span><span class="n">table</span><span class="o">,</span> <span class="n">headers</span><span class="o">)</span> <span class="n">setBody</span><span class="o">(</span><span class="n">table</span><span class="o">)</span> <span class="o">}</span> <span class="o">}</span></code></pre> tag:vy.github.io,2015-08-13://blog/post/2015/08/13/browsing-with-firefox/ Notes on Browsing the Web with Firefox 2015-08-13T18:14:00Z 2015-08-13T18:14:00Z <p>I had always been a big fan of free and open source software (F/OSS). To the point of trying to avoid using any propriatery software on systems that I have control over. My main motivatation for going along this route has many dimensions within itself that are offered in the nature of F/OSS ecosystem. That is, sources are publicly accessible, you can modify the product to adapt your needs, you can directly get in touch with developers to have any sort of information on the product internals, there is (most of the time) a strong community and user base behind the project, etc. And you know that it <em>does no evil</em>.</p> <p>Nearly 2 years ago, my frustration with Firefox hit to its peak and I completely switched to Google Chrome. From then on, no need to worry about unreasonably slow page loads, lagging scrolls, video/audio encoding issues, etc. Chrome was working perfectly fine with all the bells and whistles one would need while surfing. And honestly, I never looked back. While I was aware of Google’s <em>do no evil</em> motto is something just in the surface, I did not imagine that they could have gone to the point of <a href="https://www.privateinternetaccess.com/blog/2015/06/google-chrome-listening-in-to-your-room-shows-the-importance-of-privacy-defense-in-depth/">turning on the microphone and actively listening to the room</a>, maybe partly due to I was so satisfied with the product experience. This felt like the last drop of a massive flood of privacy invasion. Right that moment I uninstalled anything related with Google Chrome on my computer and switched to Firefox back. So how did it feel to surf the web just using F/OSS tools after 2 years? Frustratingly disappointing – particularly, while using Firefox 39.0.3 shipped with Ubuntu GNU/Linux.</p> <ul> <li> <p>You almost cannot have a proper browsing experience in <em>IMDB, Facebook, YouTube and Twitter</em>. Almost every video is not supported by the browser by default. Further, YouTube limits you to have 360p at maximum. I do not even want to talk about the annoying advertisement videos that litters all the joy. (I do not know if it is just me, but I was not shown any ads while using Chrome.)</p> </li> <li> <p>I am Turkish guy living in the Netherlands, communicating in English, and learning Dutch along the way. A majority of utility and govermental mails that I receive and websites I visit for similar purposes are all in Dutch. You cannot imagine what an impediment it was for me to live without <em>Google Translate</em> support shipped with Google Chrome.</p> </li> <li> <p>In order to study Dutch, I am regularly using <a href="https://www.duolingo.com/"><em>Duolingo</em></a>. And it was not functioning properly under Firefox.</p> </li> <li> <p>When you hit to a damn slow website, Firefox just falls down to its knees and the entire application becomes unresponsive. Compared to this, Google Chrome’s <em>one process for each tab</em> approach was working far superior and I experienced absolutely no troubles with it.</p> </li> <li> <p>Needless to say, Google Chrome saves you a <em>larger view</em> with all the buttons, fields, etc. are packed in a smaller area.</p> </li> </ul> <p>Long story short, switching back to Firefox took all the fun away from surfing. It felt like I was working in an army base, where access to internet is limited by a damn load of firewall rules. On the other hand, I also understand (some of) Firefox’s shortcomings, which were there for a certain reason like sticking to free and open standards without causing any potential licensing issues. Anyway, right now I believe I found a middle ground. I only open Google Chrome for certain websites and use Firefox for the rest. While I am not totally satisifed with this solution either, it is something I can stick to until there exists a better one.</p>