Jenkins Dynamic Locking - jenkins

I am an automation engineer, and I use Jenkins for our automated tests.
I have to test each test on multiple platforms, so a build may have these parameters;
OS (Windows 7, Windows 8, XP 64bit, XP 32bit, etc...)
Server (Server of our product, version x, version y, etc...)
Product Version (x, y, etc...)
And more...
The OS chosen determines which VM (Virtual Machine) will be used as the testing grounds.
The thing is, I have many such tests, and those who run the tests do not always check what VM is already in use, or if they set an automatic test during another automatic test's time with a specific VM.
I want the build to wait until the VM is clear to be used.
I tried to play around with the Locks and Latches plugin - changing the plugin to check each lock if it's name appears in the build parameters, and if it does, check it's value. So if the lock's name is "OS Type", and the build has the parameter "OS Type = Windows 7" it would mean the build searches for the lock "Windows 7" to see if it is free or not.
I managed to do the above part - but now when I run the tests, the first test builds it's environment, and the other tests wait for it to finish the entire build, without even checking for locks!
Thanks to that, I don't even know if what I did works.
Can anyone help? Did anyone do something like that?
I will post the code below, but as I said, I am not sure if it works as intended.
Thanks in advance!
public class LockWrapper extends BuildWrapper implements ResourceActivity {
private List<LockWaitConfig> locks;
public LockWrapper(List<LockWaitConfig> locks) {
for(LockWaitConfig lock : locks)
{
}
this.locks = locks;
}
public List<LockWaitConfig> getLocks() {
return locks;
}
public void setLocks(List<LockWaitConfig> locks) {
this.locks = locks;
}
#Override
public Descriptor<BuildWrapper> getDescriptor() {
return DESCRIPTOR;
}
#Extension
public static final DescriptorImpl DESCRIPTOR = new DescriptorImpl();
/**
* #see ResourceActivity#getResourceList()
*/
public ResourceList getResourceList() {
ResourceList resources = new ResourceList();
for (LockWaitConfig lock : locks) {
resources.w(new Resource(null, "dynamic-locks/" + lock.getName(), DESCRIPTOR.getWriteLockCount()));
}
return resources;
}
#Override
public Environment setUp(AbstractBuild abstractBuild, Launcher launcher, BuildListener buildListener) throws IOException, InterruptedException {
final List<NamedReentrantLock> backups = new ArrayList<NamedReentrantLock>();
List<LockWaitConfig> locks = new ArrayList<LockWaitConfig>(this.locks);
// sort this list of locks so that we _always_ ask for the locks in order
Collections.sort(locks, new Comparator<LockWaitConfig>() {
public int compare(LockWaitConfig o1, LockWaitConfig o2) {
return o1.getName().compareTo(o2.getName());
}
});
// build the list of "real" locks
for (LockWaitConfig lock : locks) {
NamedReentrantLock backupLock;
String varName = lock.getName();
String temp = varName;
if(abstractBuild.getBuildVariables().containsKey(varName))
{
temp = abstractBuild.getBuildVariables().get(varName).toString();
buildListener.getLogger().println("Variable " + varName + " found, replacing it with the value '" + temp + "'");
}
do {
backupLock = DESCRIPTOR.backupLocks.get(temp);
if (backupLock == null) {
DESCRIPTOR.backupLocks.putIfAbsent(temp, new NamedReentrantLock(temp));
}
} while (backupLock == null);
backups.add(backupLock);
}
final StringBuilder locksToGet = new StringBuilder();
CollectionUtils.forAllDo(backups, new Closure() {
public void execute(Object input) {
locksToGet.append(((NamedReentrantLock) input).getName()).append(", ");
}
});
buildListener.getLogger().println("[Dynamic Locks] Locks to get: " + locksToGet.substring(0, locksToGet.length()-2));
boolean haveAll = false;
while (!haveAll) {
haveAll = true;
List<NamedReentrantLock> locked = new ArrayList<NamedReentrantLock>();
DESCRIPTOR.lockingLock.lock();
try {
for (NamedReentrantLock lock : backups) {
buildListener.getLogger().print("[Dynamic Locks] Trying to get " + lock.getName() + "... ");
if (lock.tryLock()) {
buildListener.getLogger().println(" Success");
locked.add(lock);
} else {
buildListener.getLogger().println(" Failed, releasing all locks");
haveAll = false;
break;
}
}
if (!haveAll) {
// release them all
for (ReentrantLock lock : locked) {
lock.unlock();
}
}
} finally {
DESCRIPTOR.lockingLock.unlock();
}
if (!haveAll) {
buildListener.getLogger().println("[Dynamic Locks] Could not get all the locks, sleeping for 1 minute...");
TimeUnit.SECONDS.sleep(60);
}
}
buildListener.getLogger().println("[Dynamic Locks] Have all the locks, build can start");
return new Environment() {
#Override
public boolean tearDown(AbstractBuild abstractBuild, BuildListener buildListener) throws IOException, InterruptedException {
buildListener.getLogger().println("[Dynamic Locks] Releasing all the locks");
for (ReentrantLock lock : backups) {
lock.unlock();
}
buildListener.getLogger().println("[Dynamic Locks] All the locks released");
return super.tearDown(abstractBuild, buildListener);
}
};
}
public void makeBuildVariables(AbstractBuild build, Map<String,String> variables) {
final StringBuilder names = new StringBuilder();
for (LockWaitConfig lock : locks) {
if (names.length() > 0) {
names.append(',');
}
names.append(lock.getName());
}
variables.put("LOCKS", names.toString());
}
public String getDisplayName() {
return DESCRIPTOR.getDisplayName();
}
public static final class DescriptorImpl extends Descriptor<BuildWrapper> {
private List<LockConfig> locks;
/**
* Required to work around HUDSON-2450.
*/
private transient ConcurrentMap<String, NamedReentrantLock> backupLocks =
new ConcurrentHashMap<String, NamedReentrantLock>();
/**
* Used to guarantee exclusivity when a build tries to get all its locks.
*/
private transient ReentrantLock lockingLock = new ReentrantLock();
DescriptorImpl() {
super(LockWrapper.class);
load();
}
public String getDisplayName() {
return "Locks";
}
#Override
public BuildWrapper newInstance(StaplerRequest req, JSONObject formData) throws FormException {
List<LockWaitConfig> locks = req.bindParametersToList(LockWaitConfig.class, "locks.locks.");
return new LockWrapper(locks);
}
#Override
public boolean configure(StaplerRequest req, JSONObject formData) throws FormException {
req.bindParameters(this, "locks.");
locks = req.bindParametersToList(LockConfig.class, "locks.lock.");
save();
return super.configure(req, formData);
}
#Override
public synchronized void save() {
// let's remove blank locks
CollectionUtils.filter(getLocks(), new Predicate() {
public boolean evaluate(Object object) {
return StringUtils.isNotBlank(((LockConfig) object).getName());
}
});
// now, we can safely sort remaining locks
Collections.sort(this.locks, new Comparator<LockConfig>() {
public int compare(LockConfig lock1, LockConfig lock2) {
return lock1.getName().compareToIgnoreCase(lock2.getName());
}
});
super.save();
}
public List<LockConfig> getLocks() {
if (locks == null) {
locks = new ArrayList<LockConfig>();
// provide default if we have none
locks.add(new LockConfig("(default)"));
}
return locks;
}
public void setLocks(List<LockConfig> locks) {
this.locks = locks;
}
public LockConfig getLock(String name) {
for (LockConfig host : locks) {
if (name.equals(host.getName())) {
return host;
}
}
return null;
}
public String[] getLockNames() {
getLocks();
String[] result = new String[locks.size()];
for (int i = 0; i < result.length; i++) {
result[i] = locks.get(i).getName();
}
return result;
}
public void addLock(LockConfig hostConfig) {
locks.add(hostConfig);
save();
}
/**
* There wass a bug in the ResourceList.isCollidingWith,
* this method used to determine the hack workaround if the bug is not fixed, but now only needs to
* return 1.
*/
synchronized int getWriteLockCount() {
return 1;
}
}
public static final class LockConfig implements Serializable {
private String name;
private transient AbstractBuild owner = null;
public LockConfig() {
}
#DataBoundConstructor
public LockConfig(String name) {
this.name = name;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
LockConfig that = (LockConfig) o;
if (name != null ? !name.equals(that.name) : that.name != null) return false;
return true;
}
#Override
public int hashCode() {
int result;
result = (name != null ? name.hashCode() : 0);
return result;
}
}
public static final class LockWaitConfig implements Serializable {
private String name;
private transient LockConfig lock;
public LockWaitConfig() {
}
#DataBoundConstructor
public LockWaitConfig(String name) {
this.name = name;
}
public LockConfig getLock() {
if (lock == null && name != null && !"".equals(name)) {
setLock(DESCRIPTOR.getLock(name));
}
return lock;
}
public void setLock(LockConfig lock) {
this.lock = lock;
}
public String getName() {
if (lock == null) {
return name;
}
return name = lock.getName();
}
public void setName(String name) {
setLock(DESCRIPTOR.getLock(this.name = name));
}
}
/**
* Extends {#code ReentrantLock} to add a {#link #name} attribute (mainly
* for display purposes).
*/
public static final class NamedReentrantLock extends ReentrantLock {
private String name;
public NamedReentrantLock(String name) {
this.name = name;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
private static final Logger LOGGER = Logger.getLogger(LockWrapper.class.getName());
}
What I changed is basically this;
for (LockWaitConfig lock : locks) {
NamedReentrantLock backupLock;
String varName = lock.getName();
String temp = varName;
if(abstractBuild.getBuildVariables().containsKey(varName))
{
temp = abstractBuild.getBuildVariables().get(varName).toString();
buildListener.getLogger().println("Variable " + varName + " found, replacing it with the value '" + temp + "'");
}
do {
backupLock = DESCRIPTOR.backupLocks.get(temp);
if (backupLock == null) {
DESCRIPTOR.backupLocks.putIfAbsent(temp, new NamedReentrantLock(temp));
}
} while (backupLock == null);
backups.add(backupLock);
}
To clarify with another example (Thank you Peter Schuetze for bringing this up)
I am trying to run different jobs that may have the same resource (testing environment)
For this example I will have two different jobs;
Job A runs some tests on any VM I choose.
Job B runs some other test on any VM I choose.
If I choose Job A to run on VM 'Windows 7', and someone else tries to run Job B on VM 'Windows 7' after Job A started running, I want Job B to be blocked until Job A is finished.
I could have many Job A and Job B variants, each set to work on a different VM, but considering my platform matrix, it would be too much to handle.
If I want to avoid using the Locks plugin, the test list will look like that;
Job A - Windows 7 - Server A
Job A - Windows 7 - Server B
Job A - Windows 8 - Server A
Job A - Windows 8 - Server B
Job A - Windows XP x64 - Server A
Job A - Windows XP x64 - Server B
Job A - Windows XP x86 - Server A
Job A - Windows XP x86 - Server B
Job B - Windows 7 - Server A
Job B - Windows 7 - Server B
Job B - Windows 8 - Server A
Job B - Windows 8 - Server B
Job B - Windows XP x64 - Server A
Job B - Windows XP x64 - Server B
Job B - Windows XP x86 - Server A
Job B - Windows XP x86 - Server B
Please consider that in reality I have ... Around 20 jobs right now, each using more or less the same resources (Testing environments, servers, etc...)
Right now I have made it so my job list is like that;
Job A - $OS_TYPE - $SERVER - $Variable - $Another_Variable
Job B - $OS_TYPE - $SERVER - $Variable - $Another_Variable
And to make sure no resource is used at the same time by more than one job, I need the locks plugin, and I need it to accept a variable as a parameter.
If you have any further questions or a need for clarification, please feel free to ask :)

Just to recap. You modified the logs and latches plugin to accept dynamic labels. You now try to run the different tests with the same job.
Did you configure the job to run concurrently? This setting allows to instances of the same job to run in parallel.
BTW, this sounds like the typical use case for a multi-configuration project.

Related

Building Splittable DoFn (SDF) to list objects in GCS

I am trying to develop custom source for parallel GCS content scanning. The naive approach would be to loop through listObjects function calls:
while (...) {
Objects objects = gcsUtil.listObjects(bucket, prefix, pageToken);
pageToken = objects.getNextPageToken();
...
}
The problem is performance for the tens of millions objects.
Instead of the single thread code we can add delimiter / and submit parallel processed for each prefix found:
...
Objects objects = gcsUtil.listObjects(bucket, prefix, pageToken, "/");
for (String subPrefix : object.getPrefixes()) {
scanAsync(bucket, subPrefix);
}
...
Next idea was to try to wrap this logic in Splittable DoFn.
Choice of RestrictionTracker: I don't see how can be used any of exiting RestrictionTracker. So decided to write own. Restriction itself is basically list of prefixes to scan. tryClaim checks if there is more prefixes left and receive newly scanned to append them to current restriction. trySplit splits list of prefixes.
The problem that I faced that trySplit can be called before all subPrefixes are found. In this case current restriction may receive more work to do after it was splitted. And it seems that trySplit is being called until it returns a not null value for a given RestrictionTracker: after certain number of elements goes to the output or after 1 second via scheduler or when ProcessContext.resume() returned. This doesn't work in my case as new prefixes can be found at any time. And I can't checkpoint via return ProcessContext.resume() because if split was already done, possible work that left in current restriction will cause checkDone() function to fail.
Another problem that I suspect that I couldn't achieve parallel execution in DirectRunner. As trySplit was always called with fractionOfRemainder=0 and new RestrictionTracker was started only after current one completed its piece of work.
It would be also great to read more detailed explanation about Splittable DoFn components lifecycle. How parallel execution per element is achieved. And how and when state of RestrictionTracker can be modified.
UPD: Adding simplified code that should show intended implementation
#DoFn.BoundedPerElement
private static class ScannerDoFn extends DoFn<String, String> {
private transient GcsUtil gcsUtil;
#GetInitialRestriction
public ScannerRestriction getInitialRestriction(#Element String bucket) {
return ScannerRestriction.init(bucket);
}
#ProcessElement
public ProcessContinuation processElement(
ProcessContext c,
#Element String bucket,
RestrictionTracker<ScannerRestriction, ScannerPosition> tracker,
OutputReceiver<String> outputReceiver) {
if (gcsUtil == null) {
gcsUtil = c.getPipelineOptions().as(GcsOptions.class).getGcsUtil();
}
ScannerRestriction currentRestriction = tracker.currentRestriction();
ScannerPosition position = new ScannerPosition();
while (true) {
if (tracker.tryClaim(position)) {
if (position.completedCurrent) {
// position.clear();
// ideally I would to get checkpoint here before starting new work
return ProcessContinuation.resume();
}
try {
Objects objects = gcsUtil.listObjects(
currentRestriction.bucket,
position.currentPrefix,
position.currentPageToken,
"/");
if (objects.getItems() != null) {
for (StorageObject o : objects.getItems()) {
outputReceiver.output(o.getName());
}
}
if (objects.getPrefixes() != null) {
position.newPrefixes.addAll(objects.getPrefixes());
}
position.currentPageToken = objects.getNextPageToken();
if (position.currentPageToken == null) {
position.completedCurrent = true;
}
} catch (Throwable throwable) {
logger.error("Error during scan", throwable);
}
} else {
return ProcessContinuation.stop();
}
}
}
#NewTracker
public RestrictionTracker<ScannerRestriction, ScannerPosition> restrictionTracker(#Restriction ScannerRestriction restriction) {
return new ScannerRestrictionTracker(restriction);
}
#GetRestrictionCoder
public Coder<ScannerRestriction> getRestrictionCoder() {
return ScannerRestriction.getCoder();
}
}
public static class ScannerPosition {
private String currentPrefix;
private String currentPageToken;
private final List<String> newPrefixes;
private boolean completedCurrent;
public ScannerPosition() {
this.currentPrefix = null;
this.currentPageToken = null;
this.newPrefixes = Lists.newArrayList();
this.completedCurrent = false;
}
public void clear() {
this.currentPageToken = null;
this.currentPrefix = null;
this.completedCurrent = false;
}
}
private static class ScannerRestriction {
private final String bucket;
private final LinkedList<String> prefixes;
private ScannerRestriction(String bucket) {
this.bucket = bucket;
this.prefixes = Lists.newLinkedList();
}
public static ScannerRestriction init(String bucket) {
ScannerRestriction res = new ScannerRestriction(bucket);
res.prefixes.add("");
return res;
}
public ScannerRestriction empty() {
return new ScannerRestriction(bucket);
}
public boolean isEmpty() {
return prefixes.isEmpty();
}
public static Coder<ScannerRestriction> getCoder() {
return ScannerRestrictionCoder.INSTANCE;
}
private static class ScannerRestrictionCoder extends AtomicCoder<ScannerRestriction> {
private static final ScannerRestrictionCoder INSTANCE = new ScannerRestrictionCoder();
private final static Coder<List<String>> listCoder = ListCoder.of(StringUtf8Coder.of());
#Override
public void encode(ScannerRestriction value, OutputStream outStream) throws IOException {
NullableCoder.of(StringUtf8Coder.of()).encode(value.bucket, outStream);
listCoder.encode(value.prefixes, outStream);
}
#Override
public ScannerRestriction decode(InputStream inStream) throws IOException {
String bucket = NullableCoder.of(StringUtf8Coder.of()).decode(inStream);
List<String> prefixes = listCoder.decode(inStream);
ScannerRestriction res = new ScannerRestriction(bucket);
res.prefixes.addAll(prefixes);
return res;
}
}
}
private static class ScannerRestrictionTracker extends RestrictionTracker<ScannerRestriction, ScannerPosition> {
private ScannerRestriction restriction;
private ScannerPosition lastPosition = null;
ScannerRestrictionTracker(ScannerRestriction restriction) {
this.restriction = restriction;
}
#Override
public boolean tryClaim(ScannerPosition position) {
restriction.prefixes.addAll(position.newPrefixes);
position.newPrefixes.clear();
if (position.completedCurrent) {
// completed work for current prefix
assert lastPosition != null && lastPosition.currentPrefix.equals(position.currentPrefix);
lastPosition = null;
return true; // return true but we would need to claim again if we need to get next prefix
} else if (lastPosition != null && lastPosition.currentPrefix.equals(position.currentPrefix)) {
// proceed work for current prefix
lastPosition = position;
return true;
}
// looking for next prefix
assert position.currentPrefix == null;
assert lastPosition == null;
if (restriction.isEmpty()) {
// no work to do
return false;
}
position.currentPrefix = restriction.prefixes.poll();
lastPosition = position;
return true;
}
#Override
public ScannerRestriction currentRestriction() {
return restriction;
}
#Override
public SplitResult<ScannerRestriction> trySplit(double fractionOfRemainder) {
if (lastPosition == null && restriction.isEmpty()) {
// no work at all
return null;
}
if (lastPosition != null && restriction.isEmpty()) {
// work at the moment only at currently scanned prefix
return SplitResult.of(restriction, restriction.empty());
}
int size = restriction.prefixes.size();
int newSize = new Double(Math.round(fractionOfRemainder * size)).intValue();
if (newSize == 0) {
ScannerRestriction residual = restriction;
restriction = restriction.empty();
return SplitResult.of(restriction, residual);
}
ScannerRestriction residual = restriction.empty();
for (int i=newSize; i<=size; i++) {
residual.prefixes.add(restriction.prefixes.removeLast());
}
return SplitResult.of(restriction, residual);
}
#Override
public void checkDone() throws IllegalStateException {
if (lastPosition != null) {
throw new IllegalStateException("Called checkDone on not completed job");
}
}
#Override
public IsBounded isBounded() {
return IsBounded.UNBOUNDED;
}
}

Rate limiting based on user plan in Spring Cloud Gateway

Say my users subscribe to a plan. Is it possible then using Spring Cloud Gateway to rate limit user requests based up on the subscription plan? Given there're Silver and Gold plans, would it let Silver subscriptions to have replenishRate/burstCapacity of 5/10 and Gold 50/100?
I naively thought of passing a new instance of RedisRateLimiter (see below I construct a new one with 5/10 settings) to the filter but I needed to get the information about the user from the request somehow in order to be able to find out whether it is Silver and Gold plan.
#Bean
public RouteLocator myRoutes(RouteLocatorBuilder builder) {
return builder.routes()
.route(p -> p
.path("/get")
.filters(f ->
f.requestRateLimiter(r -> {
r.setRateLimiter(new RedisRateLimiter(5, 10))
})
.uri("http://httpbin.org:80"))
.build();
}
Am I trying to achieve something that is even possible with Spring Cloud Gateway? What other products would you recommend to check for the purpose if any?
Thanks!
Okay, it is possible by creating a custom rate limiter on top of RedisRateLimiter class. Unfortunately the class has not been architected for extendability so the solution is somewhat "hacky", I could only decorate the normal RedisRateLimiter and duplicate some of its code in there:
#Primary
#Component
public class ApiKeyRateLimiter implements RateLimiter {
private Log log = LogFactory.getLog(getClass());
// How many requests per second do you want a user to be allowed to do?
private static final int REPLENISH_RATE = 1;
// How much bursting do you want to allow?
private static final int BURST_CAPACITY = 1;
private final RedisRateLimiter rateLimiter;
private final RedisScript<List<Long>> script;
private final ReactiveRedisTemplate<String, String> redisTemplate;
#Autowired
public ApiKeyRateLimiter(
RedisRateLimiter rateLimiter,
#Qualifier(RedisRateLimiter.REDIS_SCRIPT_NAME) RedisScript<List<Long>> script,
ReactiveRedisTemplate<String, String> redisTemplate) {
this.rateLimiter = rateLimiter;
this.script = script;
this.redisTemplate = redisTemplate;
}
// These two methods are the core of the rate limiter
// Their purpose is to come up with a rate limits for given API KEY (or user ID)
// It is up to implementor to return limits based up on the api key passed
private int getBurstCapacity(String routeId, String apiKey) {
return BURST_CAPACITY;
}
private int getReplenishRate(String routeId, String apiKey) {
return REPLENISH_RATE;
}
public Mono<Response> isAllowed(String routeId, String apiKey) {
int replenishRate = getReplenishRate(routeId, apiKey);
int burstCapacity = getBurstCapacity(routeId, apiKey);
try {
List<String> keys = getKeys(apiKey);
// The arguments to the LUA script. time() returns unixtime in seconds.
List<String> scriptArgs = Arrays.asList(replenishRate + "", burstCapacity + "",
Instant.now().getEpochSecond() + "", "1");
Flux<List<Long>> flux = this.redisTemplate.execute(this.script, keys, scriptArgs);
return flux.onErrorResume(throwable -> Flux.just(Arrays.asList(1L, -1L)))
.reduce(new ArrayList<Long>(), (longs, l) -> {
longs.addAll(l);
return longs;
}) .map(results -> {
boolean allowed = results.get(0) == 1L;
Long tokensLeft = results.get(1);
Response response = new Response(allowed, getHeaders(tokensLeft, replenishRate, burstCapacity));
if (log.isDebugEnabled()) {
log.debug("response: " + response);
}
return response;
});
}
catch (Exception e) {
/*
* We don't want a hard dependency on Redis to allow traffic. Make sure to set
* an alert so you know if this is happening too much. Stripe's observed
* failure rate is 0.01%.
*/
log.error("Error determining if user allowed from redis", e);
}
return Mono.just(new Response(true, getHeaders(-1L, replenishRate, burstCapacity)));
}
private static List<String> getKeys(String id) {
String prefix = "request_rate_limiter.{" + id;
String tokenKey = prefix + "}.tokens";
String timestampKey = prefix + "}.timestamp";
return Arrays.asList(tokenKey, timestampKey);
}
private HashMap<String, String> getHeaders(Long tokensLeft, Long replenish, Long burst) {
HashMap<String, String> headers = new HashMap<>();
headers.put(RedisRateLimiter.REMAINING_HEADER, tokensLeft.toString());
headers.put(RedisRateLimiter.REPLENISH_RATE_HEADER, replenish.toString());
headers.put(RedisRateLimiter.BURST_CAPACITY_HEADER, burst.toString());
return headers;
}
#Override
public Map getConfig() {
return rateLimiter.getConfig();
}
#Override
public Class getConfigClass() {
return rateLimiter.getConfigClass();
}
#Override
public Object newConfig() {
return rateLimiter.newConfig();
}
}
So, the route would look like this:
#Component
public class Routes {
#Autowired
ApiKeyRateLimiter rateLimiter;
#Autowired
ApiKeyResolver apiKeyResolver;
#Bean
public RouteLocator theRoutes(RouteLocatorBuilder b) {
return b.routes()
.route(p -> p
.path("/unlimited")
.uri("http://httpbin.org:80/anything?route=unlimited")
)
.route(p -> p
.path("/limited")
.filters(f ->
f.requestRateLimiter(r -> {
r.setKeyResolver(apiKeyResolver);
r.setRateLimiter(rateLimiter);
} )
)
.uri("http://httpbin.org:80/anything?route=limited")
)
.build();
}
}
Hope it saves a work day for somebody...

TinyIoC Returning Same instance

I am new to the dependency injection pattern and I am having issues getting a new instance of a class from container.Resolve in tinyioc it just keeps returning the same instance rather than a new instance. Now for the code
public abstract class HObjectBase : Object
{
private string _name = String.Empty;
public string Name
{
get
{
return this._name;
}
set
{
if (this._name == string.Empty && value.Length > 0 && value != String.Empty)
this._name = value;
else if (value.Length < 1 && value == String.Empty)
throw new FieldAccessException("Objects names cannot be blank");
else
throw new FieldAccessException("Once the internal name of an object has been set it cannot be changed");
}
}
private Guid _id = new Guid();
public Guid Id
{
get
{
return this._id;
}
set
{
if (this._id == new Guid())
this._id = value;
else
throw new FieldAccessException("Once the internal id of an object has been set it cannot be changed");
}
}
private HObjectBase _parent = null;
public HObjectBase Parent
{
get
{
return this._parent;
}
set
{
if (this._parent == null)
this._parent = value;
else
throw new FieldAccessException("Once the parent of an object has been set it cannot be changed");
}
}
}
public abstract class HZoneBase : HObjectBase
{
public new HObjectBase Parent
{
get
{
return base.Parent;
}
set
{
if (value == null || value.GetType() == typeof(HZoneBase))
{
base.Parent = value;
}
else
{
throw new FieldAccessException("Zones may only have other zones as parents");
}
}
}
private IHMetaDataStore _store;
public HZoneBase(IHMetaDataStore store)
{
this._store = store;
}
public void Save()
{
this._store.SaveZone(this);
}
}
And the derived class is a dummy at the moment but here it is
public class HZone : HZoneBase
{
public HZone(IHMetaDataStore store)
: base(store)
{
}
}
Now since this is meant to be an external library I have a faced class for accessing
everything
public class Hadrian
{
private TinyIoCContainer _container;
public Hadrian(IHMetaDataStore store)
{
this._container = new TinyIoCContainer();
this._container.Register(store);
this._container.AutoRegister();
}
public HZoneBase NewZone()
{
return _container.Resolve<HZoneBase>();
}
public HZoneBase GetZone(Guid id)
{
var metadataStore = this._container.Resolve<IHMetaDataStore>();
return metadataStore.GetZone(id);
}
public List<HZoneBase> ListRootZones()
{
var metadataStore = this._container.Resolve<IHMetaDataStore>();
return metadataStore.ListRootZones();
}
}
However the test is failing because the GetNewZone() method on the Hadrian class keeps returning the same instance.
Test Code
[Fact]
public void ListZones()
{
Hadrian instance = new Hadrian(new MemoryMetaDataStore());
Guid[] guids = { Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid() };
int cnt = 0;
foreach (Guid guid in guids)
{
HZone zone = (HZone)instance.NewZone();
zone.Id = guids[cnt];
zone.Name = "Testing" + cnt.ToString();
zone.Parent = null;
zone.Save();
cnt++;
}
cnt = 0;
foreach (HZone zone in instance.ListRootZones())
{
Assert.Equal(zone.Id, guids[cnt]);
Assert.Equal(zone.Name, "Testing" + cnt.ToString());
Assert.Equal(zone.Parent, null);
}
}
I know its probably something simple I'm missing with the pattern but I'm not sure, any help would be appreciated.
First, please always simplify the code to what is absolutely necessary to demonstrate the problem, but provide enough that it will actually run; I had to guess what MemoryMetaDataStore does and implement it myself to run the code.
Also, please say where and how stuff fails, to point others straight to the issue. I spent a few minues figuring out that the exception I was getting was your problem and you weren't even getting to the assertions.
That said, container.Resolve<HZoneBase>() will always return the same instance because that's how autoregistration in TinyIoC works - once an abstraction has been resolved, the same instance is always returned for subsequent calls.
To change this, add the following line to the Hadrian constructor:
this._container.Register<HZoneBase, HZone>().AsMultiInstance();
This will tell the container to create a new instance for each resolution request for HZoneBase.
Also, Bassetassen's answer about the Assert part is correct.
In general, if you want to learn DI, you should read Mark Seemann's excellent book "Dependency Injection in .NET" - not quite an easy read as the whole topic is inherently complex, but it's more than worth it and will let you get into it a few years faster than by learning it on your own.
In your assert stage you are not incrementing cnt. You are also using the actual value as the expected one in the assert. This will be confusing, becuase it says something is excpected when it actually is the actual value that is returned.
The assert part should be:
cnt = 0;
foreach (HZone zone in instance.ListRootZones())
{
Assert.Equal(guids[cnt], zone.Id);
Assert.Equal("Testing" + cnt.ToString(), zone.Name);
Assert.Equal(null, zone.Parent);
cnt++;
}

Accessing a Service from within an XNA Content Pipeline Extension

I need to allow my content pipeline extension to use a pattern similar to a factory. I start with a dictionary type:
public delegate T Mapper<T>(MapFactory<T> mf, XElement d);
public class MapFactory<T>
{
Dictionary<string, Mapper<T>> map = new Dictionary<string, Mapper<T>>();
public void Add(string s, Mapper<T> m)
{
map.Add(s, m);
}
public T Get(XElement xe)
{
if (xe == null) throw new ArgumentNullException(
"Invalid document");
var key = xe.Name.ToString();
if (!map.ContainsKey(key)) throw new ArgumentException(
key + " is not a valid key.");
return map[key](this, xe);
}
public IEnumerable<T> GetAll(XElement xe)
{
if (xe == null) throw new ArgumentNullException(
"Invalid document");
foreach (var e in xe.Elements())
{
var val = e.Name.ToString();
if (map.ContainsKey(val))
yield return map[val](this, e);
}
}
}
Here is one type of object I want to store:
public partial class TestContent
{
// Test type
public string title;
// Once test if true
public bool once;
// Parameters
public Dictionary<string, object> args;
public TestContent()
{
title = string.Empty;
args = new Dictionary<string, object>();
}
public TestContent(XElement xe)
{
title = xe.Name.ToString();
args = new Dictionary<string, object>();
xe.ParseAttribute("once", once);
}
}
XElement.ParseAttribute is an extension method that works as one might expect. It returns a boolean that is true if successful.
The issue is that I have many different types of tests, each of which populates the object in a way unique to the specific test. The element name is the key to MapFactory's dictionary. This type of test, while atypical, illustrates my problem.
public class LogicTest : TestBase
{
string opkey;
List<TestBase> items;
public override bool Test(BehaviorArgs args)
{
if (items == null) return false;
if (items.Count == 0) return false;
bool result = items[0].Test(args);
for (int i = 1; i < items.Count; i++)
{
bool other = items[i].Test(args);
switch (opkey)
{
case "And":
result &= other;
if (!result) return false;
break;
case "Or":
result |= other;
if (result) return true;
break;
case "Xor":
result ^= other;
break;
case "Nand":
result = !(result & other);
break;
case "Nor":
result = !(result | other);
break;
default:
result = false;
break;
}
}
return result;
}
public static TestContent Build(MapFactory<TestContent> mf, XElement xe)
{
var result = new TestContent(xe);
string key = "Or";
xe.GetAttribute("op", key);
result.args.Add("key", key);
var names = mf.GetAll(xe).ToList();
if (names.Count() < 2) throw new ArgumentException(
"LogicTest requires at least two entries.");
result.args.Add("items", names);
return result;
}
}
My actual code is more involved as the factory has two dictionaries, one that turns an XElement into a content type to write and another used by the reader to create the actual game objects.
I need to build these factories in code because they map strings to delegates. I have a service that contains several of these factories. The mission is to make these factory classes available to a content processor. Neither the processor itself nor the context it uses as a parameter have any known hooks to attach an IServiceProvider or equivalent.
Any ideas?
I needed to create a data structure essentially on demand without access to the underlying classes as they came from a third party, in this case XNA Game Studio. There is only one way to do this I know of... statically.
public class TestMap : Dictionary<string, string>
{
private static readonly TestMap map = new TestMap();
private TestMap()
{
Add("Logic", "LogicProcessor");
Add("Sequence", "SequenceProcessor");
Add("Key", "KeyProcessor");
Add("KeyVector", "KeyVectorProcessor");
Add("Mouse", "MouseProcessor");
Add("Pad", "PadProcessor");
Add("PadVector", "PadVectorProcessor");
}
public static TestMap Map
{
get { return map; }
}
public IEnumerable<TestContent> Collect(XElement xe, ContentProcessorContext cpc)
{
foreach(var e in xe.Elements().Where(e => ContainsKey(e.Name.ToString())))
{
yield return cpc.Convert<XElement, TestContent>(
e, this[e.Name.ToString()]);
}
}
}
I took this a step further and created content processors for each type of TestBase:
/// <summary>
/// Turns an imported XElement into a TestContent used for a LogicTest
/// </summary>
[ContentProcessor(DisplayName = "LogicProcessor")]
public class LogicProcessor : ContentProcessor<XElement, TestContent>
{
public override TestContent Process(XElement input, ContentProcessorContext context)
{
var result = new TestContent(input);
string key = "Or";
input.GetAttribute("op", key);
result.args.Add("key", key);
var items = TestMap.Map.Collect(input, context);
if (items.Count() < 2) throw new ArgumentNullException(
"LogicProcessor requires at least two items.");
result.args.Add("items", items);
return result;
}
}
Any attempt to reference or access the class such as calling TestMap.Collect will generate the underlying static class if needed. I basically moved the code from LogicTest.Build to the processor. I also carry out any needed validation in the processor.
When I get to reading these classes I will have the ContentService to help.

db4o Tranparent Persistence doesn't store later objects in my own ActivatableCollection<T>

I'm rolling my own ActivatableCollection<T> for db4o but cribbing heavily from the builtin ActivatableList<T> implementation. I'm running into the problem where transparent persistence doesn't seem to be working correctly. In the test code below:
[Fact]
void CanStoreActivatableCollection()
{
var planets = new ActivatableCollection<Planet>();
var pagingMemoryStorage = new PagingMemoryStorage();
var config = Db4oEmbedded.NewConfiguration();
config.Common.Add(new TransparentActivationSupport());
config.Common.Add(new TransparentPersistenceSupport());
config.File.Storage = pagingMemoryStorage;
var objectContainer = Db4oEmbedded.OpenFile(config, "Memory.yap");
planets.Add(new Planet("Mercury"));
objectContainer.Store(planets);
planets.Add(new Planet("Venus"));
planets.Add(new Planet("Earth"));
objectContainer.Commit();
objectContainer.Close();
config = Db4oEmbedded.NewConfiguration();
config.Common.Add(new TransparentActivationSupport());
config.Common.Add(new TransparentPersistenceSupport());
config.File.Storage = pagingMemoryStorage;
objectContainer = Db4oEmbedded.OpenFile(config, "Memory.yap");
planets = objectContainer.Query<ActivatableCollection<Planet>>().FirstOrDefault();
Assert.NotNull(planets);
Assert.Equal(3, planets.Count);
objectContainer.Close();
}
The planet "Mercury" is stored, but not "Venus" and "Earth". If I change from ActivatableCollection to ActivatableList, then all 3 planets are stored.
What am I missing? My ActivatableCollection is just minimal implementation of ActivatableList as best as I can tell.
Below is my implementation of ActivatableCollection:
public class ActivatableCollection<T>
: ICollection<T>
, IActivatable
, INotifyCollectionChanged
{
List<T> _list;
List<T> List
{
get
{
if (_list == null)
_list = new List<T>();
return _list;
}
}
public ActivatableCollection()
{
}
public int Count
{
get
{
ActivateForRead();
return List.Count;
}
}
public bool IsReadOnly
{
get
{
ActivateForRead();
return ((IList) List).IsReadOnly;
}
}
public void Add(T t)
{
ActivateForWrite();
List.Add(t);
OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Add, t));
}
public void Clear()
{
ActivateForWrite();
List.Clear();
OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Reset));
}
public bool Contains(T t)
{
ActivateForRead();
return List.Contains(t);
}
public void CopyTo(T[] array, int index)
{
ActivateForRead();
List.CopyTo(array, index);
}
public IEnumerator<T> GetEnumerator()
{
ActivateForRead();
return List.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
public bool Remove(T t)
{
ActivateForWrite();
bool removed = List.Remove(t);
if (removed)
OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Remove, t));
return removed;
}
[Transient]
private IActivator _activator;
public virtual void Bind(IActivator activator)
{
if (_activator == activator)
return;
if (activator != null && _activator != null)
throw new InvalidOperationException();
_activator = activator;
}
public virtual void Activate(ActivationPurpose purpose)
{
if (_activator == null)
return;
_activator.Activate(purpose);
}
protected virtual void ActivateForRead()
{
Activate(ActivationPurpose.Read);
}
protected virtual void ActivateForWrite()
{
Activate(ActivationPurpose.Write);
}
[Transient]
public event NotifyCollectionChangedEventHandler CollectionChanged;
protected virtual void OnCollectionChanged(NotifyCollectionChangedEventArgs e)
{
if (CollectionChanged != null)
CollectionChanged(this, e);
}
}
I've also tried copying the code from GenericTypeHandlerPredicate and registering my ActivatableCollection to use the GenericCollectionTypeHandler. That results in a crash in GenericTypeFor() throwing an InvalidOperationException() when "Mercury" is being stored.
Just want to mention my answers from the db4o forums also here, for people with a similar problem:
First part of the issue:
From db4o's point of view nothing has changed in the 'ActivatableCollection' object and therefore no changes are stored. This is what is happening:
When you add the items, the ActivatableCollection is marked as changed.
When you commit the changes are stored. However the ' ActivatableCollection' holds the reference to the same object. db4o only stores the changes in the ActivatableCollection-object, which is the reference to the List. Since it is the same, no actual change is stored.
The List of the ActivatableCollection is never updated, because it wasn't marked as 'changed'
So the transparent activation doesn't see the changes in the list. You can fix your issue simply by using an ActivatableList in you're ActivatableCollection implementation. Just change the List with a IList interface and instantiate a ActivatableList instead of an List.
The second part of the issue: Why doesn't it work even when registering the GenericCollectionTypeHandler for this type? Here we hit a implementation detail. The GenericCollectionTypeHandler has an internal list of supported types, which doesn't include the self made 'ActivatableCollection'. GenericCollectionTypeHandler is not really part of the public API and intendet for internal use only.
Workaround / Fix
Just use an ActivatableList<T> instead of a List<T>. then everything works fine.

Resources