Talk:Fuzzing

Open source cited as inheriently insecure
Acoording to this article, open-source software is inheritently less secure than closed-source software -- this is quite surprising since one of the major claims of the open source enthutiasts is that open source is more secure than closed source.

The problem is the following sentence: Since major customer and enterprise management software is starting to be open-source, database-based security attacks are becoming more credible.

Whomever wrote that, should clarify why, being open source, this is so.

After having read the sentence again, I think it should also be clarified what exactly is ment -- are open source database systems less secure or are open source databases (systems?) used somehow in attacking other systems? And more importantly, just why is this realted to fuzzing at all? FrederikHertzum (talk) 14:32, 25 September 2008 (UTC)


 * Oddly, there is a security connection between open source software and fuzzing, but it is not what you might expect. Fuzzing treats the software as a "black box", in which none of the internal workings (including source code) are visible. Because fuzzing is so effective at finding glitches without source code and because some of the earliest fuzzing experiments showed the FSF's GNU suite of Unix tools to be more robust than any proprietary equivalents, open-source proponents sometimes cite fuzzing as yet another reason people should not rely on the false security through obscurity provided by closed-source software. I'm not sure about adding this to the article, however. While this information is citable, I'm not sure that it has encyclopedic worth. Thoughts? Ben (talk) 06:05, 19 August 2014 (UTC)

Clarification of intro needed
I clicked on the "fuzz testing" link in a user page (it read "I've been fuzz testing MediaWiki"). Although I'm far from a technological illiterate, the opening paragraph still doesn't quite make the meaning clear to the layperson. My questions are as follows: I know that one of the Wikipedia guidance documents mentions that there's only so far you can water down some very technical subjects, and that some articles are likely only to be accessed by people who already have some background in the terminology and concepts of the field. However, I think that this introduction is trying hard to use simple terms, is currently unsuccessful in that attempt, and is very close to success. Lawikitejana 02:32, 7 September 2006 (UTC.)
 * The basic idea is to attach the inputs of a program ... Given that software is essentially intangible, it is unclear how one would "attach" anything. Additionally, it's unclear what "the inputs" means. I know the term "input," but can't make sense of it in this usage.
 * to a source of random data ("fuzz") ... need an example or clarification there, too, though it may become more self-evident once "attach the inputs" is cleared up.
 * (for example, ... by failing built-in code assertions) ... I submit that a typical layperson doesn't know what "code assertions" are.

Credit
The introduction to the article appears to give credit for the idea of fuzz testing to Barton Miller and students in 1989. Yet at the bottom of the page, there is a link to folklore.org describing how fuzz testing was being done on the Mac on 1983! It seems that fuzz testing might have originated independently several times, perhaps with the Wisconsin group being the first to publish it, at least in a major journal (CACM, I think?). It might be more appropriate to have a "history" section on the page which discusses these multiple origins, instead of giving all the credit to one group.

I agree with the dubiousness of this being created in 1989; I know that I personally used it in 1985, and I didn't invent it at that time, as I had heard of others doing it. Not sure if it was published, but certainly it was already widely known in 1985. - Glenn


 * Precisely, someone started a page on fuzzing history. It has been unchanged since 2008 but seems correct. The contents of this page should be merged into the Fuzz testing page, and the Mac 1983 folklore.org you mention should also be added. 188.111.75.64 (talk) 18:17, 1 May 2013 (UTC)

Actually, George J. Carrette, the author of the 1991 UNIX fuzzer "crashme", cites his design as coming from the formal specification in Cybernetics by Norbert Wiener. That book was published in 1948, so it would seem to have precedence if credit is to be given to anyone. Here is the quote from the crashme webpage: ''A bit of background on crashme. It is a tool for testing the robustness of an operating environment using a technique of "Random Input" response analysis. This I first saw formally proposed in the book Cybernetics by Norbert Wiener, but which any parent who has observed his children playing and learning would be well disposed to describe in detail.'' I am not an expert in the field, so I will not edit the page directly, but I think it would make sense to at least make mention of Norbert Wiener and his book. And while the true originator of "fuzzing" may never be found I suggest changing this sentence in the article's introduction: The field of fuzzing originated with Barton Miller at the University of Wisconsin in 1988. It is pretty clear that, while the term fuzzing may be from 1988, the field of fuzzing has been around a lot longer than that. Perhaps, if Carrette is right, fuzzing originated when the first child played with the first toy by poking at it to see what it would do. Ben (talk) 05:48, 19 August 2014 (UTC)

Glitching?
why the two way links between Glitching and fuzz testing. i fail to see the connection. —The preceding unsigned comment was added by 189.172.38.108 (talk) 22:22, 10 February 2007 (UTC).

"There are at least two different forms of fuzz testing"
Article says "There are at least two different forms of fuzz testing:" and proceeds to show 3 bullets.

The last bullet should be outside the list, or the text above should say "...three..." —Preceding unsigned comment added by 61.11.49.253 (talk) 11:55, 28 December 2007 (UTC)

Mutation analysis
I think whoever wrote this article didn't know what mutation analysis was. It's not comparable to fuzz testing at all, as M.A. is for testing the test suite, not (directly) for finding bugs. I've deleted the whole paragraph discussing differences between the two. --Povman (talk) 00:32, 21 May 2008 (UTC)

Company marketing
I´ve kicked out MuDynamics company marketing because of unnecessary blabla. They should improve their systems before they go and make marketing for low price fuzzers that are only available on an apliance that is not scalable or otherwise working good. They have focus on a very small band of protocols and they cannot be used on mobile things (would you carry an appliance weighing multi pounds to test a wireless access point - hahahaha). - Anonymous

But shouldn't there be an overview of the Fuzz testing products available? For example, I would add gremlins.js, which does this for JS apps. - Dsernst (talk) 13:15, 17 July 2014 (UTC)

More marketing
i propose to remove this text since it's not really related to this topic. this page is allegedly about fuzz testing then this section talks about how fuzz testing isn't good enough (are value judgments germane?) and goes on to promote a particular company's technology:


 * "Robustness testing" was introduced by PROTOS researchers in 1999 (most of them now part of Codenomicon, headquartered in Oulu, Finland) to increase the test efficiency through systematic tests. Robustness testing is a model based fuzzing technique, an extension of syntax testing, that systematically will explore the input space defined by various communication interfaces or data formats, and will generate intelligent test cases that find crash-level flaws and other failures in software.

i really don't see why this belongs on this page at all; go create a page on robustness testing and say whatever is relevant to that subject on that page —Preceding unsigned comment added by T0pgear09 (talk • contribs) 07:53, 28 March 2009 (UTC)

How long shall I wait for comments on the proposal above?
if i dont hear back by 35 march (ok, i mean 4 april lol :) i'll go ahead and do what i suggested; if someone objects i guess they can revert the change (but i still think they should have to make a convincing argument to put it back since it seems off-topic for this page) T0pgear09 (talk) 05:35, 30 March 2009 (UTC)

Ok i did it
i feel like the text i removed stoodd out since it was about some alternative to the subject of the page but that description boiled down to a commercial for a company, not a description of the technology that probably should be on its own page anyway, not the anti-page within this page ;) T0pgear09 (talk) 04:26, 4 April 2009 (UTC)

Peacock terms removed
thuis pg is in my watchlist and so i noted recent changes; other than removing Peacock terms per WP:PEACOCK and re-wording a bit seems ok to me

i cant helep but wondeer if hte person means this org in finland: http://www.ee.oulu.fi/research/ouspg/ which is not exactly what is referred to in the text provided but maybee the name the person wrote is the orig name? come to think of it, i think the mention of a company in this line is outside the bounds of an ext link and i;ll do another edit to remove just that bit and ill chng the name of the group to whats on the website —Preceding unsigned comment added by T0pgear09 (talk • contribs) 05:55, 1 May 2009 (UTC)

added back a reference and a clarification
I don't visit here often but I just caught up on edits over the last year. Maybe the Network Statgey Partners reference I added back was originally removed by mistake? It now has a home near all the other links to commercial papers on fuzzing. Informationh0b0 (talk) 17:28, 30 September 2009 (UTC)

Random testing != fuzz testing
The computer science litterature does not equate random testing with fuzz testing. Only Wikipedia does this. In general, fuzzing is a particular form of (black-box or white-box) random testing.

See, e.g., Patrice Godefroid, Michael Y. Levin, and David Molnar SAGE: Whitebox Fuzzing for Security Testing Communications of the ACM, 2012. http://research.microsoft.com/en-us/um/people/pg/public_psfiles/cacm2012.pdf which defines "blackbox fuzzing" as "a form of blackbox random testing, which randomly mutates well-formed program inputs and then tests the program with those modified inputs".

Yet, Wikipedia redirects "random testing" to "fuzz testing". Because Wikipedia takes things in the other way round, the explanations are nothing but confusing.

Vasywriter (talk) 19:37, 8 April 2013 (UTC)


 * The fact there is a redirect between random testing and fuzz testing is not an endorsement that they two terms mean the same thing. I suggest that you look at the history of that "article" to determine how that happened. Since the term is not frequently used in the software development world and since no articles point there, it's not really an issue. If you actually want to resolve the issue, we can nominate that article for deletion. Walter Görlitz (talk) 20:26, 8 April 2013 (UTC)

I can't follow your point. Which term is not frequently used in the software development world? Random testing is universally used. Fuzz testing is also intensively used (cf. Microsoft SAGE, just for one instance). 188.111.75.64 (talk) 18:03, 1 May 2013 (UTC)

As of today, I see that the "random testing" page is separate, and includes a link to here. But it is also important to clarify in this article what makes "Fuzz testing" different than "random testing" - as my impression from reading the article was that it was just random testing. On the other hand the explanation by Vasywriter indicates that it does have a distinct meaning. AmirOnWiki (talk) 11:50, 5 February 2014 (UTC)

Actually computer science literature does not provide any clear distinction between monkey testing, random testing and fuzz testing. The older (and trivial) term is Monkey testing, the most accepted one is Random testing, and finally the more popular one these days in CS is Fuzz testing, but this just a term popularity. The particular paper cited by Vasywriter actually only describes one randomization strategy, that their authors think it a definition of fuzzing does not make it one. OWASP for instance lists a much wider variety of randomization strategies under Fuzzing, including "for binary: random ones", cf. https://www.owasp.org/index.php/Fuzzing 89.158.120.107 (talk) 07:30, 18 March 2016 (UTC)
 * Then computer science literature should get updated. Software testing literature often provides clear distinctions. Check StickyMinds.com Walter Görlitz (talk) 17:37, 18 March 2016 (UTC)

Heartbleed, other fuzzing tools and the fuzzing project
Maybe it would be a good idea to implement this information into the article. Here a few links:
 * https://blog.hboeck.de/archives/868-How-Heartbleed-couldve-been-found.html
 * https://fuzzing-project.org/
 * http://www.golem.de/news/fuzzing-wie-man-heartbleed-haette-finden-koennen-1504-113345.html (German)

Some tools which are mentioned (and could be added to the article) are: American Fuzzy Lop, Address Sanitizer --rugk (talk) 19:21, 27 April 2015 (UTC)
 * Those sources for Heartbleed are not RSes.
 * We would have to decide if we only wanted to include notable tools (i.e., those with articles) or open it up to any tool that claims to be a fuzz testing tool. Walter Görlitz (talk) 03:55, 28 April 2015 (UTC)


 * The sources are reliable. They are all from an author of a more or less well-known German news website (third link) which is golem.de. Golem.de was also used in quite many English Wikipedia articles as a source. --rugk (talk) 19:48, 28 April 2015 (UTC)
 * They are? I don't think a blog is a RS, more-or-less well-know is less. Feel free to take it to WP:RSN though.
 * That we have a single author pushing the idea is also problematic.
 * You didn't discuss whether we wanted to include notable tools (i.e., those with articles) or open it up to any tool that claims to be a fuzz testing tool. Walter Görlitz (talk) 04:33, 29 April 2015 (UTC)
 * Golem.de is well-known (in 2012 it was on the 5th place of websites with focus on technical topics). If or what tools to include I leave up for others to decide.
 * BTW also the Linux Foundation now supports the fuzzing project:
 * * http://www.golem.de/news/open-source-linux-foundation-bewilligt-geld-fuer-weitere-security-projekte-1506-114833.html
 * * http://www.linuxfoundation.org/news-media/announcements/2015/06/linux-foundation-s-core-infrastructure-initiative-funds-three-new
 * --rugk (talk) 08:41, 28 June 2015 (UTC)


 * How about tools that have been specifically discussed in the course of academic research? Another option is creating a tools comparison entry. MikeAntares (talk) 22:37, 16 May 2016 (UTC)
 * e.g., this paper on fuzzing, with a table comparing tools on page 33. — Preceding unsigned comment added by MikeAntares (talk • contribs) 22:34, 17 May 2016 (UTC)

Merge Monkey testing into Fuzz testing
Monkey testing appears to just be another form of Fuzz testing. TheGuyOfDoom (talk) 13:10, 23 January 2016 (UTC)
 * First, the article is at monkey test, the noun form, not the verb form.
 * Second, that article has a source, and several articles that use the term.
 * I see no compelling reason as to why they should not be merged. Walter Görlitz (talk) 18:08, 23 January 2016 (UTC)
 * Both should be merged, but given that monkey testing is the older terminology, it might make more sense to merge the other way (fuzz testing is just a new name of monkey testing) 89.158.120.107 (talk) 07:10, 18 March 2016 (UTC)
 * Not true. They are clearly different terms. Monkey testing is random manual testing. Fuzz testing is testing where an algorithm is in control of flow of testing and is programmatic. Walter Görlitz (talk) 17:37, 18 March 2016 (UTC)
 * Do we have any source that can clearly distinguish between the two? The articles themselves suggest a merger is due. François Robere (talk) 18:44, 10 April 2016 (UTC)
 * No source clearly states that they are the same thing. We would need a source that states the one is a type of the other not that a source states they are different. Walter Görlitz (talk) 02:01, 11 April 2016 (UTC)
 * My point is that the articles themselves suggest they are the same thing, so barring new sources there's no reason not to merge them; the other option is to rewrite one or both of them so that they're distinct. Mergers are at our discretion if the content is identical but the naming is different, which is the case here.
 * As for sources, see here: François Robere (talk) 17:14, 12 April 2016 (UTC)
 * Stack overflow is not a RS and I can't read Hebrew so I can't confirm what it says. There is a superficial resemblance between the two subjects, but monkey testing also has other connotations and that is not the same as fuzz testing. Walter Görlitz (talk) 04:20, 13 April 2016 (UTC)
 * Hebrew wasn't the point there, but rather the book You can read the forward here: . Note who wrote it.
 * SO is often used as a source here, nevertheless, as many writers there are very capable and knowledgeable.
 * Now, why are we at all deliberating about merging a stub? With only a single, 18 y/o source it barely has any reason for existence, let alone separately of fuzz testing. François Robere (talk) 13:24, 13 April 2016 (UTC)
 * It's not the same topic. See below. Walter Görlitz (talk) 14:11, 13 April 2016 (UTC)


 * Merge to fuzz testing - the two articles appear to be describing the same thing, and this article already has a couple of paragraphs about the 1983 "Monkey" Mac OS testing application. Fuzz testing seems to be the more commonly-used term. --McGeddon (talk) 13:41, 13 April 2016 (UTC)
 * That there was a program called Monkey on the Mac does not mean it was known as Monkey testing. Spreadsheets are not known as 1-2-3 because the first program was called that. I could go on. Walter Görlitz (talk) 14:11, 13 April 2016 (UTC)
 * It's just striking that both articles mention "monkeys" and the infinite monkey thought experiment. But please do go on; I don't follow your "random manual testing" and "monkey testing also has other connotations" arguments above at all, from reading the monkey test article alone. --McGeddon (talk) 16:14, 13 April 2016 (UTC)
 * I concur. François Robere (talk) 17:09, 13 April 2016 (UTC)
 * Go on? Sure. p34 and following in Cem Kaner's presentation: http://kaner.com/pdfs/highvolCSTER.pdf (and other locations). It also has been equated to testing on the fly (or ad hoc testing). So it's not a one-size-fits-all definition. Walter Görlitz (talk) 04:51, 14 April 2016 (UTC)


 *  Merge Monkey Test to Fuzz Testing. According to this source, monkey testing is just one method of fuzz testing.   Scr ★ pIron IV 17:27, 13 April 2016 (UTC)
 * A reference I can sink my teeth into. Thanks. Walter Görlitz (talk) 04:51, 14 April 2016 (UTC)
 * Don't forget to brush! François Robere (talk) 11:55, 15 April 2016 (UTC)
 * I agree with ScrapIronIV. There are fuzzing techniques that are very different from monkey testing. Monkey Testing is just another term for Random Testing and as a blackbox testing technique does not require any knowledge about the structure of the tested program. However, there are also symbolic execution-based whitebox fuzzers that do heavily utilize program structure to reach deeper parts of the tested program.Ikkeer (talk) 07:25, 8 March 2017 (UTC)
 * Looks like there's been a debate about whether they are the same thing, and I've rewritten the Monkey test article to reflect the debate as well as current sources. I think it'll be good to keep them separately as there won't be consensus any time soon. Aaroniidx (talk) 02:03, 22 April 2016 (UTC)


 * Support - It seems clear that the two are closely related. The structure of the two articles is now similar so it doesn't look like a merge would be difficult. ~Kvng (talk) 16:44, 3 May 2016 (UTC)


 * According to the article, the term "monkey" was used in 1983 to describe automated random input generation at Apple. Also according to the article, the term "fuzzing" was coined in 1988, and describes the simplest form of monkey testing. Noel Nyman wrote at least three articles about testing with dumb monkeys in which he described a range of capabilities of the monkeys. "Smart monkeys" use some information to qualify the random inputs and/or check some characteristics of the outcomes. The first article was Much more recently the term "fuzz testing" has been used by security testers. Since "fuzz testing" is a new term for a subset of tests that have been known as "dumb monkey tests" for years, maybe "fuzz testing" belongs here as a footnote (or as an alternate term for the simplest monkey). — Preceding unsigned comment added by Doug.hoffman (talk • contribs) 05:10, 20 May 2016 (UTC)


 * Merge A load of cack. Fashion driven naming for established testing technique. Merge. 22:12, 5 January 2017 (UTC)


 * Don't Merge - Merge consensus is that articles are describing the same thing but they are not the same. Monkey testing is just one fuzzing technique, specifically it is generation-based, random, blackbox fuzzing for programs with an interactive user interface.
 * Monkey testing is a generation-based blackbox fuzzing technique that generates random inputs (see Fuzzing). Nyman mentions that the "monkey" broadly refers "to any form of automated testing done randomly and without any 'typical user' bias." It would make more sense to merge monkey testing and random testing.
 * Monkey testing is the fuzzing of programs with graphical user interfaces. Nyman mentions that "smart monkeys have some knowledge about how to access the user interface in the product they’re testing". The well-known Android Monkey "is a program that [..] generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events."
 * Monkey testing is one kind of fuzzing. Brummayer et al. state that "fuzz testing methods such as 'monkey testing' were already used around 1980."
 * Ikkeer (talk) 13:38, 14 March 2017 (UTC)


 * Deadline 28.06.17 - The last item was added 3 months ago after a major revision of the article. If there is no discussion until 28.06.17, I feel it is reasonable to remove the request to merge from the page. Ikkeer (talk) 02:50, 21 June 2017 (UTC)
 * There are no deadlines, but if you want to make it June 17, 2028, I'm fine with that. Don't make any changes before that (but others are welcome to of course). Cheers. Walter Görlitz (talk) 03:32, 21 June 2017 (UTC)
 * Not quite sure what you mean. You reverted my removal of the proposal tag on May 24, stating "no deadline". Just checked: The Wiki page on mergers reads "If there is [..] no consensus or no discussion and you don't believe it is appropriate to merge the pages, then please remove the merge proposal tags, and, if necessary, close any discussion." So, I'll go ahead an remove the proposal tag. Not sure how to close the discussion. Ikkeer (talk) 13:24, 28 June 2017 (UTC)

External links modified
Hello fellow Wikipedians,

I have just added archive links to 1 one external link on Fuzz testing. Please take a moment to review my edit. If necessary, add after the link to keep me from modifying it. Alternatively, you can add to keep me off the page altogether. I made the following changes:
 * Added archive http://web.archive.org/web/20110718122610/http://wurldtech.com/resources/SB_002_Robustness_Testing_With_Achilles.pdf to http://wurldtech.com/resources/SB_002_Robustness_Testing_With_Achilles.pdf

When you have finished reviewing my changes, please set the checked parameter below to true or failed to let others know (documentation at ).

Cheers.—cyberbot II  Talk to my owner :Online 03:19, 29 February 2016 (UTC)

Further reading / PDF link
A PDF of the the last article in the list exists at http://langsec.org, which I linked to in the last change (presently reverted as "Copyright violation. © 2014 IEEE and the site is not IEEE"). As far as I can tell, this is the author-posted "accepted version" of the article at his web page. (Compare https://www.ieee.org/documents/author_faq.pdf Sec. 2.) That this is indeed the author's web page can be verified (at present, this may change) by visiting http://www.cs.dartmouth.edu/~sergey/langsec/ and observing that it's identical to http://langsec.org, including the remark in the lower left corner. ("Please link to this page as http://langsec.org/.")

As such, the copyright violation claim should be bogus. Unless further complaints are voiced, I will attempt to revert the revert once in a few hours or so. 2A02:F28:1:D:50B9:DD7C:5B4B:5485 (talk) 01:35, 21 November 2016 (UTC)
 * Since IEEE articles are not written by individuals, but by committee, no single author can be identified. As such you claim is invalid. IEEE holds the copyright to all of its documents. Walter Görlitz (talk) 04:08, 21 November 2016 (UTC)

Ain't nobody says Fuzz testing
This article should be named Fuzzing, as that is the de-facto name used in the security community. I do not have a proper reference for this, other than if you do a Google search for "fuzzing" you get ~800k hit, and ~300k for "fuzz testing". Also the results on the later page often mention "fuzzing" as an alias, whereas the former do not mention "fuzz testing". I will go ahead and make the required changes, if you disagree, let's have a discussion before reverting. — SkyLined (talk) 08:15, 22 December 2016 (UTC)
 * The move was not immediately possible. I've created a request here Requested_moves/Technical_requests
 * I endorse this move. In my experience as a software test automation engineer, this type of testing is most often referred to as "fuzzing." As that is the common name, it should be in the article title. Alex Eng ( TALK ) 09:02, 22 December 2016 (UTC)
 * Support. Although a Google search for "fuzzing" also returns usage of the term in several other contexts, appending "software testing" to the search seems to go about 2:1 in favour of "fuzzing", both on a Google and Google Books search. --McGeddon (talk) 09:19, 22 December 2016 (UTC)

Rewriting
I am restructuring this article. I am a senior research fellow and have been working on the topic of automated software testing for my PhD and well into my postdoctoral time. I've always felt that the page on fuzzing does not reflect the recent contributions and viewpoints of security researchers and practitioners. The article was clearly unorganized and convoluted with scattered discussions and topics of arbitrary importance. So, I opened an account just for the purpose of rewriting this article. I am new to Wikipedia, so feel free to clean up syntactically or to remind me if I happen to violate some policy. For instance, I am not very sure how to reference (PDFs of) conference or journal papers. If you have any suggestions for interesting topics to discuss, please leave them below. E.g., Ikkeer (talk) 13:48, 11 March 2017 (UTC)
 * 1) Adding the Heartbleed and other incidents that could be found via fuzzing.
 * 2) Adding the uses of fuzzers as vulnerability detection tool, together with ASAN or other memory debuggers. Small discussion of oracle problem.
 * 3) If we decide to merge, then monkey testing would be merged into fuzzing. All monkey testers are fuzzers, but not all fuzzers are monkey testers.
 * That's all good, but only if the new content is well sourced. Walter Görlitz (talk) 16:35, 11 March 2017 (UTC)
 * Sure. Let me know where you'd like additional citations. Ikkeer (talk) 00:46, 12 March 2017 (UTC)

Add Fuzz Stati0n to the list of companies offering cloud based fuzzing
I added Fuzz Stati0n to the list of companies offering cloud based fuzzing (in addition to Google and Microsoft) but the edit was reverted.

To justify the edit: Fuzz Stati0n was founded on July 16th, 2016 as a Delaware LLC. It was announced at a meetup on Sept 7th, 2016 that Fuzz Stati0n is offering a cloud based fuzzing service and that was covered in the press (please see the sidebar.)

To justify the order, Microsoft announced Project Springfield on Sept 26, 2016.

Fuzz Stati0n has also had more press and won an award.

Disclaimer: I am the founder of the company.

Thank you,

Grajagandev (talk) 19:51, 7 September 2017 (UTC)