How are heaps (for example, the one implemented by typical
malloc routines, or by Windows's
HeapCreate, etc.) implemented, typically? What data structures do they use?
While searching online, I've seen tons of descriptions of how to implement heaps with severe restrictions.
To name a few, I've seen lots of descriptions of how to implement:
And it's funny, they all avoid the harder question:
How are "normal", general-purpose heaps (like the one behind
What data structures (and perhaps algorithms) do they use?
Note: The following answer assumes you're using a typical, modern system with virtual memory. The C and C++ standards do not require virtual memory; therefore of course you can't rely on such assumptions on hardware without this feature (e.g. GPUs typically don't have this feature; nor do extremely small hardware like the PIC).
This depends on the platform you're using. Heaps can be very complicated beasts; they don't use only a single data structure; and there is no "standard" data structure. Even where the heap code is located is different depending on the platform. For instance, the heap code is typically provided by the C Runtime on Unix boxes; but is typically provided by the operating system on Windows.
sbrk). Instead of returning memory to the operating system, many heaps only try to reuse memory no longer in use by the program proper, and don't try to return memory to the system. This is less common on Windows, because its equivalent to
VirtualAlloc) doesn't have this limitation. (But like
sbrk, it is very expensive and has caveats like only allocating page-sized and page-aligned chunks. So heaps try to call either as rarely as possible)
RtlHeapmaintains a number of data structures like this for different known block sizes. (E.g. it'll have one for blocks of size 16, for instance) RtlHeap calls these "lookaside lists".
The best reference I've found discussing the common allocation strategies employed on major platforms is the book Secure Coding in C and C++, by Robert Seacord. All of chapter 4 is dedicated to heap data structures (and problems caused when users use said heap systems incorrectly).
What are some tips I can use to avoid memory leaks in my applications? In my current project I use a tool "INSURE++" which finds the memory leak and generate the report.
Apart from the tool is there any method to identify memory leaks and overcome it.
There are three main ways of doing this.
The first is to not create memory leaks in the first place. Defensive programming techniques are invaluable here. See this excellent presentation for a summary of this issues, or the relevant chapter in Secure C Coding. I am more familiar with C than C++, but I understand that C++'s smart pointers are useful here.
A second approach static analysis, which attempts to detect errors in your source-code. The original tool in this category is lint, which is now sadly outdated. The best tools, as far as I know, are commercial such as coverty. However, some free tools do exist.
The third approach is to detect memory leaks at runtime, like INSURE++ does. Valgrind is excellent here and highly recommended. It may help catch bugs you've already introduced. It is especially helpful if you do have a test suite that has good code coverage.
I know this is vague question but I'm looking for blogs or information regarding how teams are integrating security with continuous delivery / deployment. We are deploying to AWS multiple times per day and I am looking for some ways teams are adding security to the flow.
I've seen one presentation where a team used cucumber to do some nmap testing, that isn't exactly what I am looking for but maybe some automated testing of app nodes once they have been deployed before they go into the load balancer accepting traffic.
This may not be what you're looking for, but the key to effective security testing is building it into the product at design time, implementation, etc. Coding security tests just like you would unit tests, at all levels of the application. Using this approach, security testing is no different than general application testing.
The pre-packaged security tests are good, and you should use them (most orgs do this just before the QA check), but they are not as effective as your built-in tests. This is because no one knows the security "danger-zones" like your developers (or at least they should. If they don't have them read a book. For web apps I highly recommend "The Web Application Hacker's Handbook," and for other apps I recommend "Secure Coding in C and C++" by Robert Seacord, even if you don't do C/C++. There's a 2nd Edition of Seacord's book coming out in April if you can wait).
Security will never be effective unless considered at design time. If you've screwed that up already, try to get security tests integrated into your regular app tests.
Some great pre-canned scanners (some free-as-in-speech, others free-as-in-beer, and others not free at all) to run against your web app (in no particular order). These will find common and existing vulnerabilities, but will not find unique vulns to your web app:
does someone know good literature for C/C++? language: german/english
This one has an entire chapter on the subject.